Showcase images and animationsPosted by Daddy-o May 23, 2013 10:47AM
Here are some images of the telescope assembled - perhaps it is possible to use the images to understand which plug goes where? Sharper jpegs are in /home/pth/SCIENCEPROJECTS/EARTHSHINE/JPEGS/ - they are the 'imgXXX.jpg' files.
I've also added images from the construction in Lund - those images are in same place as above, but are called imagexxx.jpg (xxx from 332 to 347).
shows more images from the telescope on MLO.
Here is an image of the cabling at the back of the CCD, during MLO.
Real World ProblemsPosted by Daddy-o May 23, 2013 09:30AMLargest box:
Telescope tube plus its attached cables.
Rack of electronics.
Very long and thick bundle of cables.
Rod for Hohlraum source + footSmaller Wood box:
LCD monitor + VGA cable
Laser printer Samsung ML-2525
Small white box: Spare lamps + relay
Holhraum Sphere in its own box
Box of fan filters for rack
2 x Shutter boards + Edmund pack of flat soft things + a small mw laser + a filter labelled 'IR cut'
European power supply for Axis camera
2 Axis cameras with robofucus lenses, and additional lenses and mounting brackets.
Rain and light sensor - DMI?
Väisala device in 'chineese tower' affair.
DMI IR rain sensor
Väisala humidity sensor HMT100
Cables + spare PXI parts + bracket for adjusting SKE
Uniblitz shutter # VS25ST1-100
Vincent Associates 710P Shutter Interconnect Cable
3 Thor labs FW drivers / electronics. Model FW102B. Implies that we have
three Thor labs FWs - the colour FW, the ND FW, and one more - somewhere!
Box of mixed cables - VGA + DVI + 2xUSB repeaters
2 sealed metal envelopes with "D.C.D. Panel Mount SA 4 COMP"
Tools for shutter tuning
Spare Ferrorperm Knife edges on glass.
Mixed VGA, DVI, USB cables along with 2 USB extender cables
US 115V power cables - to PC-type plugs.Blue box:
Polar alignment scope
Point source lamp + scope
Magma + mounting HW
Dome breakout box
base w. electronics
Power for Axis camera (perhaps it moved to Wooden box?9
A part of the PXI - probably the spare or original PC
Spare HD for PXIWhere are:
Mount handbox controller?
Aurora cloud sensor?
Various interface cards for camera to PXI?Additional
Real World ProblemsPosted by Daddy-o May 23, 2013 08:34AM
Here is a collection of links useful in our attempts to power up the rack of control system, at DMI:
iBootBar : http://dataprobe.com/support/index.html
Optical designPosted by Daddy-o May 21, 2013 07:58AM
Here are example sof a 'ghost' and 'dragging':
The dragging is due to no shutter being used - i.e. readout was only way to terminate exposure and hence frame was illuminated as the image was shifted to the 'hidden register' We hav a frame-transfer camera. The image is from the test phase in Lund and no shutter was installed then.
The BS ghost is faintly vissible: It is a copy of the BS and the lower cusp is seen poking out slightly down and to the left of the real BS. This was due to the CCD camera being aligned almost perfectly along the optical axis of the system - the ghost arises as the shiny CCD surface reflects light back into the optical system and reflections are produced at all optical surfaces - the back of the second secondary lens, the front of the second secondary lens, the back of the first secondary lens, the front of the first secondary lens and the back of the primary objective and its various surfaces. At MLO the camera was tilted at an angle so that the reflection from the CCD did not go back up the system. This tilting did not appear to have any adverse effects on focus etc.
Images from lab hard disc recovered from MLO.
To-do listPosted by Peter Thejll May 20, 2013 12:39PM
We are starting a new blog over at
where issues related to how to design (and one day, hopefully, build) a better eshine telescope are considered.
The present blog will from now on only discuss how to use the data obtained from the Hawaii telescope.
From flux to AlbedoPosted by Daddy-o May 06, 2013 03:42PM
Some of the notes discussed below, in the next several postings, are collected in this doc:
There are some considerations of how how much albedo change we expect during global warming - and the detectability of such changes are discussed.
From flux to AlbedoPosted by Daddy-o May 06, 2013 09:17AM
But we - and BBSO - calculate albedo by comparing the earthshine measured on the Moon to the intensity of earthshine predicted by a terrestrial model based on a uniform Lambert sphere.
We test how well this works by taking a series of GERB satellite data for several weeks across a year and extract the total flux from the whole-disk images. The MSG satellite bearing the GERB instrument floats over lon,lat=0,0 so always sees the same part of Earth. Johanne has extracted images for every fifteen minutes for several weeks in a year. We plot that (top panel, below).
We use the eshine synthetic code Hans wrote to generate Lambertian images for a full cycle of sunrise over earth. As the code is based on what the Earth looks like from the Moon we pretend that one month is like one day and thereby can extract the phase law for the Lambertian uniform-albedo Earth in order to compare it to the GERB data. We also plot that (second panel, below). [Note that the difference between view from satellite and view from Moon may be important: Sun-Earth-Viewpoint angles should be the same, and if the Sun and Moon are not at similar latitudes as Sun and Satellite we could be generating artefacts in what follows: we should see if we can use the synthetic code for satellite viewpoints. For now we ought to find dates when the Moon was at latitude 0 (like the satellite) and the Sun also same latitudes - tricky to do.
Lastly we divide Gerb fluxes by Lambertian fluxes, correct for the fact that Geostationary orbit and the Moon are at different distances and multiply that corrected ratio by the uniform albedo used in the Lambertian models. We plot that (bottom panel, below).
We see that the albedo does not come out constant. This is not surprising since the Earth has real clouds that drift around - but that is only what gives the thickness of the thick line of points in the last plot above. The 'wiggles' are due to the inadequacy of the Lambertian model. Near New Earth (Full Moon: never observed) the derived albedo rises. Near Full Earth (New Moon: attractive to observe due to strong eshine, but difficult due to Moon close to Sun) the albedo is flat. At intermediary values (20% and 80% of the cycle) the Lambertian albedo is relatively high so that the derived albedo is lowered.
How can we use these insights to understand what Johanne shows in plots of how derived albedo evolves during the nights?
Our aspiration is that the above can give insight into
1) the 'phase dependency' we see in derived albedos when we plot all data corresponding to all phases during the morning branch - i.e. Moon setting over Western Pacific/Australia, and
2) the nightly tendency to have falling albedo through the night, for that same branch.
As for 1 the reader should look in the May 3 presentation at slide 22; as for 2 the reader shoul dlook at slide 23.
We have to figure out whther the above plots explain any of these sightings. Could the almost quadratic phase dependency seen when all data are plotted be due to the 'dip' near 20 and 80% of the cycle? Could the 'nightly slopes' be due to the same?
Real World ProblemsPosted by Peter Thejll May 05, 2013 02:31PM
Our efforts to measure albedo precisely and accurately are limited by the natural variability. Albedo is inherently a quite variable property of the Earth - viewed day-to-day [see presentation linked to below for details on the GERB data variability!]. Just how much does global-mean albedo vary - in the short run and over many years?
We have CERES albedo data for the period 2000 to 2012. It is given as monthly mean data for a 360x180 degree grid. We take this and calculate global-mean monthly values. Some results are here:Data downloaded from: http://ceres.larc.nasa.gov/cmip5_data.php
these are the monthly-mean data prepared by NASA for the CMIP5 effort.
I calculate the global weighted means for the given monthly-mean values
of upward and incoming shortwave light at TOA.
The linfit slope is:
Slope: 5.4920926e-08 +/- 5.7758642e-07 in albedo units/day.
This is an INsignificant slope - per decade it would amount to 0.09 %
of the mean albedo. Using, instead of the slope, the +/- errors on the slope we get:
+/- 0.97 % per decade.
The mean we get is:
Mean albedo: 0.218214
this is small. Publications using these data say: 0.29 ... there is some
problem with accounting for regions' weights. I omit all areas where the
incoming flux at TOA is less than 12 W/m² - this helps avoid Inf's and
NaN's. I weight each area with the cosine of the latitude of cell middle.
Month Albedo S.D. S.D. as % of mean
1 0.228720 +/- 0.000820013 or, +/- 0.358522 %.
2 0.222061 +/- 0.000866032 or, +/- 0.389997 %.
3 0.206976 +/- 0.000856246 or, +/- 0.413694 %.
4 0.215791 +/- 0.000886397 or, +/- 0.410766 %.
5 0.219209 +/- 0.00129653 or, +/- 0.591460 %.
6 0.219878 +/- 0.00122390 or, +/- 0.556628 %.
7 0.215800 +/- 0.000900811 or, +/- 0.417429 %.
8 0.212036 +/- 0.000893605 or, +/- 0.421440 %.
9 0.203454 +/- 0.000899703 or, +/- 0.442214 %.
10 0.215973 +/- 0.00123064 or, +/- 0.569814 %.
11 0.227462 +/- 0.000582274 or, +/- 0.255988 %.
12 0.231050 +/- 0.00111927 or, +/- 0.484429 %.
What do we learn?
We learn that albedo is remarkably constant when observed by satellite. There is no discernible slope to the data but if we use the 1 sigma uncertainties on the slope as upper limits we find that per decade the albedo has changed less than 1%.
The monthly means follow an understandable annual cycle (maxima in NH and SH winters with minima in March and September). The spread around monthly means amount to 0.25 to 0.6% of the monthly mean value.
Climatologically it is an open question whether albedo ought to change with climate drift. During the observing period global mean surface temperatures have changed by about +0.1 degree C [see http://www.ncdc.noaa.gov/sotc/service/global/global-land-ocean-mntp-anom/201101-201112.png ]. This is during the 'hesitation period' that is much discussed presently. During other decades mean T has risen much more - but we have no albedo data from these periods.
Using the above 1 sigma upper limits on slope of 1% per decade we see that if the slope is due to changes in T then the relationship is 1% per 0.1 degree or 10% per degree. This is based on an upper limit and the true value is closer to 0% per degree.
Note that the previous argument is unrelated to the EBM based relationship that works for equilibrium climate only - there, and only there, the expected relationship is -1% per degree.
So, what does that give us? If we were to observe a larger slope we could use the data in the "satellites are getting it wrong" mode - as Pallé et al did for a while. If we measure no slope we can hope to set more stringent limits to the slope than the above satellite values do - can we determine the slope of global mean albedo to better than the 1% per decade above? In this
presentation I found upper limits of 0.2% per decade based on a null hypothesis of 'no albedo change' and realistic observing limitations. The numbers used were based on Frida Benders CERES data, but they do not differ enormously from the present, longer, ones.
So can we reach 0.2% error per decade, observationally?
This requires a discussion of the single-frame errors we get as well as the period-mean data we can expect. More alter!
Showcase images and animationsPosted by Chris Flynn May 05, 2013 05:36AM
Presentation at Swinburne (Melbourne) in November 2012
Showcase images and animationsPosted by Daddy-o May 03, 2013 12:31PM
This earthshine presentation was given at DMI, on May 3 2013.
Post-Obs scattered-light rem.Posted by Daddy-o Apr 29, 2013 10:29AM
We have previously
considered B-V images of the Moon. This was done with 'raw' images - that is, images where the halo had not been removed. Since we have the BBSO linear method implemented and since it does clean up the DS we can also calculate B-V images for the Moon based on these.
We have 55 pairs of B and V images thata re close in time (about 1 houror less apart). Using the standard star calibration relationships that Chris worked out fropm NGC6633 standard stars, we can convert images to instrumental magnitud eimages and from there to calibrated B and V magnitud eimages. We also corrected for extinction since the images were not obtained at the same time.
SInce the calibration relationships depend on B-V we have to assume some B-V values and iterate (Chris solves algebraically). The iterations converge quickly. We use onlythe brightest pixels in each image - i.e. the pixels delineating the BS - for calculating the mean B and mean V values needed to update B-V in each iteration.
The values for BS B-V that we converge to have this distribution:
The mean B-V=0.989, and the S.D.=0.019. The accepted value - e.g. Allen (4th ed), Table 12.14, gives B-V=0.92 (van den Bergh observations?). We therefore have a significant discrepancy. It should probably be noted that our values come from phases near 90 degrees, while the Allen values may be from 'Full Moon' conditions.
If we accept the above B-V (BS) values at face value we can continue:
The Sun has B-V=0.642 (Holmberg et al, MNRAS 2006). One reflection off the Moon reddends this value by 0.989 - 0.642 = 0.347. This value will also apply to earthshine that is observed after one reflection off the Moon, even if it is the DS. [We ignore here any colour-dependencies in the lunar surface mare vs highlands!
If we can estimate the B-V of earthshine as seen on the lunar surface, we can work backwards to what the B-V of that light was before it struck the Moon - it will be the observed value minus 0.347.
Before trying this we need to understand to which degree the use of BBSO linear images, as opposed to 'raw' images, has helped us observe the true colour of the DS - has an important amount of the BS halo been removed from the DS?
We generate centered B-V images and plot the average of 20 rows across the middle of the images:We see two panels - each panel is the result of using a fixed B image and two different V images - all three taken a short time apart. The black curve is the run of B-V values in the 'raw' image - that is, the image where no effort has been made to remove the BS halo. The red curve is from images cleaned with the BBSO linear method. The deep jag in the middle is the BS/DS terminator. The DS is to the left of this and the BS to the right. Since the BS is not altered by the BBSO linear method the red curve covers the black curve on the BS.
On the DS we see that cleaning the image has resulted in a slight reddening of the DS - it was 'too blue' in the red images.
We also see that the 'linear gradient' in B-V across the DS is unaltered qualitatively by the cleaning of the image. Why?
If we push on, ignoring the not-yet-understood gardient, and assume that the part of the DS closest to the sky has an un-polluted B-V value then we can calculate the colour of earthshine before it strikes the Moon, as explained above. First we extracted DS B-V values for that part of the DS disk that is to the left of 90% of the vertical columns on the disk. These values were on average 0.29 +/- 0.05 below the BS value.
If the BS value is given the canonical B-V=0.92, then we have a B-V for the DS of 0.63.
Franklin (JGR 72, no 11, p.2963-, 1967) measured B and V repeatedly on the DS. The difference between his mean B and his mean V is 0.64. We are close, but we are worried about scattered light!
Subtracting the effect of reflection once on the Moon brings us to the value for B-V of earthshine, before it strikes the Moon, that is, as it would be seen in space:
B-V_ES = 0.28.
There is one published B-V value for earthshine, based on Mariner II data in the 1960s. The paper is http://adsabs.harvard.edu/abs/1964JGR....69.4661W by Wildey. Unfortunately I cannot make head or tail of that paper!
Playing a bit more with the above, we can consider the effect of Earth on light - the Sun has B-V=0.642 when it strikes Earth. If the earthshine has B-V=0.28, then the bluing effect of Earth is 0.28-0.642 = -0.36 in B-V.
Real World ProblemsPosted by Daddy-o Apr 24, 2013 10:34AM
We update yet again the list of 'best images' that Chris has generated by inspection of compliance of absolute magnitudes against lunar phase.
We can removea few more images by hand inspection. We found about 10 that have 'cable in view' as well as various near-horizon problems. The list is here and now contains 525 images:
We note that Johanne is working her way through many images and finding 'bad focus' cases. Since we believe these are coincident with 'not the right filter acquired' cases, we shall eventually be further updating the list of best images.
Data reduction issuesPosted by Peter Thejll Apr 17, 2013 04:52PM
We are using co-add mode - that is, we take stacks of 100 images and co-add them to increase the SNR. Since there is RON (read Out Noise) in each frame - about 2 ADU/pixel, tests have shown - it is important to know whether a sequence of added frames have a mean that converges.
We now test this directly on the stack of 100 images: "2456015.7742682MOON_V_AIR". We extract a small square of size (2*w+1) around regularly spaced points covering the image plane and calculate a mean subframe (averaging along the stack direction), the average value of that mean subframe (a scalar for each of n coadded subframes), and plot these. These are shown in the PDF file:
The top panel in each page shows where the point was chosen. w was 4 so subframes are 9 pixels by 9 pixels, centred on the point. Second panel in each page shows how the mean value of the average subframes evolves, along with estimates of the +/-1 sigma error bounds (calculated based on the value the series converges to, or at least evolves to, in the last 10 steps of the series; the known RON (about 2.4 ADU/pixel in all observations, estimated from bias frames differences), the number of pixels in each subframe and the number of subframes co-added). Last panel shows the mean value of each sub-frame used (along the stack direction).
What do we see? In page 1 a point on the sky has been selected and we see a mean value that evolves inside the expected error bounds, and we see that the subframes have mean values with some spread, but no trend.
Page 11 shows a point on the DS. Nice evolution.
Page 24 is on the BS and is very hard to explain. This holds also for points 'near the BS'.
It seems we have little to worry about on the sky and on the DS, but on the BS we see strange evolution of the running mean! We have before touched on such subjects in this blog, when we tested effects of alignment of images - perhaps this problem was hiding inside the other problem?[added later:]
Chris asked a good question. Here is a plot of the running mean of the whole frame, as frames are taken from the stack.
The value is expressed as a percentage deviation from the middle value on the curve - about 1378 counts. A small drift is seen at the 0.05% level from one frame at left to all frames at right. I think this could be due to sky variations - or slight drift of the Moon inside the frame causing 'light to fall off the edge'. If drift is the explanation we may have an answer for why the mean in a much smaller sub-frame, when near the BS, drifts so much more - image brightness gradients are being sampled in a small subframe. Experiment at top should be redone but with drifts taken out. No large drift was evident sp I guess we are learning that a small drift can be very important!
This has an implication for how we measure the albedo - if we use 'DS patch divided by BS patch' we run a risk . if we use 'DS patch divided by total flux' we are much better off.
Post-Obs scattered-light rem.Posted by Chris Flynn Apr 07, 2013 09:38AM
The PSF of the VE2-band Arcturus data deviated markedly from the other bands (next to previous post). We check on this further by looking at the PSF profiles of the stars in NGC6633, which were used to calibrate the filters (in this post).
We plot the central flux normalised PSFs in B, VE1, VE2 and IRCUT versus V (shown in green in each panel). The PSF is formed from 95 different stars spread over the frame in each case: e.g. this shows the stars in the IRCUT image.
The PSFs look like this:
Same plot as above but in log-linear form:
None of the PSFs are as sharp as the V band one -- they all have more
flux at greater distance from the center. Interestingly, the VE2 band
profile is not discrepant, so the conclusion is that the IRCUT data for
Arcturus may simply have been out of focus.
It would seem the lesson from this is that the PSF can vary quite a bit -- but note well that this is in the core only -- as this is not a test of the extended power-law wings at all, as we are only probing to a radius of about 4 pixel (~30 arcsec) with these data.
Post-Obs scattered-light rem.Posted by Chris Flynn Apr 07, 2013 03:59AM
In the previous post, we noticed that the VE1 and IRCUT filters have very similar profiles to the V band -- at least for the data we obtained pointing at Arcturus.
Here we show a colour map (V-VE1, on an arbitrary magnitude scale: properly calibrated colours will come later).
It's clear that the deep artifacts around the edge of the moon are much smaller now, compared to the B-V colour maps we have been producing to date.
This shows that the core of the PSF of the V and VE1 filters are quite similar -- and the power law tails as well -- not just for Arcturus but for lunar images too.
Post-Obs scattered-light rem.Posted by Chris Flynn Apr 07, 2013 12:53AM
This is a followup to the previous post on the V and B band PSFs.
Here is the stellar profile of Arcturus, shown in B (blue) and V (green). Clearly the B band light has a different core to the PSF than V.
Arcturus data were taken on the night of 2011-03-22. The files used are
Oddly, the V band data are affected by what we called "shutter bounce" -- a non-axisymmetric feature to the right (along crows in the CCD) of the PSF, but the B band data show no sign of such a bounce (the PSF profile can be exceedingly cleanly centered, unlike V band). No explanation for this for the moment!
Here are the other bands, all compared to V (green symbols):
PDF version here:
Here are the same profiles, this time with the data averaged into radial bins. The solid green line is the V band profile in each case.
VE1 and V are very similar -- but this is what we would expect. IRCUT is not too far off V either.. the B and in particular the VE2 filters differ significantly from V.
The difference between VE2 and all the other filters is huge, so we need to look into that next!
Post-Obs scattered-light rem.Posted by Chris Flynn Apr 06, 2013 01:22AM
It is clear from the colour maps we have been making of the moon, that the B band PSF must be quite a bit broader than the one in V-band. We have only measured the V-band profile with any precision to date: this is a first go at the B-band profile.
The images used are these:
both taken at an airmass of ~1.3 - so quite high in the sky (which is good).
A wedge shaped region is defined for each, as shown in this figure by the green vectors:
The (V band) image is on a logarithmic scale, with counts/pixel shown along bottom. The red circle marks the fitted (by eye) position of the moon.
Centers and radii are :
V band center (195, 287), radius 147 pix
B band center (197, 291), radius 147 pix
Plot shows log radius (from center of moon, in pixels) versus flux (counts/pixel) in V band (green curve) and B band (blue curve).
It's clear that the dark side of the moon (DS) is bluer than the bright side (blue curve offset by ~0.05 in the log on the DS).
Plotting distance from the edge of the bright side, we get this:
Focusing on the falloff of light off the brightside edge (coincidentally the B and V band fluxes are almost identical in the two images, so they are effectively normalised at the peak flux) -- we see in the figure above that the blue light has considerably broader scattered light than the V band. (Note that the radial scale is now linear, not log).
Next plot shows the falloff of light on a log-log scale, as a function of distance from the lunar edge (less 3 pixels, in order to catch maximum flux).
Clearly blue light is more scattered than V!
Up to now, we have been using a single V-band scattering profile for the light -- which is based on this technique of measuring the scattered light off the lunar edge, fitting a power law to the falloff at large distance (the power law tail of the profile, which has a slope of order -2.7 to -2.9 on very clear nights, at least in the V band).
The power law tail in B appears to have the same slope as in V -- the B band light beyond ~ 10^1.5 = 30 pixels from the lunar edge out to the limit of the wedge (250 pixels) follows V band closely.
However, there is a large excess of light from the lunar edge out to about 25 pixels. This is messing up our colour maps, producing the dark (very blue) artifacts around the moon.
More work is certainly needed! A deconvolution method to extract the true PSF from full moon images, in all our bands, might be the thing to try next, rather than the "poor man's" wedge method! (Note that in the wedge method, we correct for the fact that the moon is not a point source, we do not use the profile of the light in the wedge directly!)
From flux to AlbedoPosted by Peter Thejll Mar 29, 2013 03:21PM
] We have discussed many ways of extracting albedo from our data. We first considered the 'ratio over ratio' methods - they consisted of extracting counts from patches on areas of the scattered-light cleaned images in the DS and the BS - the ratio of DS/BS observed to DS/BS model is proportional to the albedo of the Earth. The other method, used more recently, is based on 'profile fitting' near the DS sky edge.
We have now arrived at comparing the 5-color albedos derived from these two methods:
Right: Albedos for positive and negative lunar phases (Full Moon is at 0 degrees) from the DS/BS method where the "BBSO log" method has been used to remove scattered light. Left: Same but for the profile fitting method. Note different
vertical scales on axis. The same nights were considered for both
plots, but not all are present in both, due to outlier removal, etc.
We see, for both methods, a rise in the derived albedo as phases nearer Full Moon are considered. This is possibly due to effects of scattered light from the BS which has been incompletely handled by the respective methods. The values found with the two methods are quite similar - apart from increased scatter and less colour-separation in the DS/BS method. The data for the positive and negative branch of the phase diagram are not similar, in either case.
From tests shown elsewhere in this blog we do not expect the halo to interfere with the DS for large absolute lunar phases - i.e. near new Moon. The above diagrams shows lowest values for phases near 110 degrees. There is a slight increase in values larger than this - what can that be due to, if the halo is interfering less and less? Well, we must again remember that the above results are model-dependent and the Model may be adding its own fingerprint. For instance, it may be that the synthetic images we use have a phase-dependent error in their representation of lunar and/or terrestrial albedos. Note that the same synthetic models are used in the DS/BS method as in the profile-fitting method.
This question can be addressed by studies of the effect on the 'bend' seen above of different BDRF models.
Bias and Flat fieldsPosted by Peter Thejll Mar 23, 2013 06:24PM
We show here the B-V mean value of all pixels lit by sunshine. The data have been selected for being 'good', so the scatter is somewhat disappointing. Best one can say is that we do not contradict the published B-V=0 .92 value.
Blue and red points correspond to different parts of the lunar cycle - before and after Full Moon - waxing and waning, whatever you want to call it.
Student projectsPosted by Daddy-o Mar 22, 2013 09:51AM
We have often come across good ideas for student projects. Here is a start of a collection of projects - just links, but text can be added to explain more.
How do meteorological conditions determine seeing at the telescope?
Was the bias pattern
Understanding the PSF:
Albedo maps and their use in modelling observations:
Atmospheric turbulence studied via Moon images:
Colour of earthshine - Danjons work:
Image analysis methods - Laplacian method:
Exploring the PSFPosted by Daddy-o Mar 22, 2013 09:30AM
post we saw that the difference between B and V (magnitude) images could have the shape of a linear slope on the DS and plateau on the BS. We are trying to recreate that using synthetic models. It is surprisingly difficult!
Using V and V images we saw that differences typically had the shape of level offsets - not slopes. In the B-V images we saw linear slopes on the BS. I thought the linear slopes originated in different PSFs in two filters - different alfa-parameters, for instance.
Well, taking a synthetic image and convolving it twice with two slightly different PSFs and converting to magnitudes and subtracting gives this:
Upper panel shows the ideal image we are using - BS to the right and the rest is DS. Bottom panel shows the difference between the image convolved with alfa=1.73 and alfa=1.72*1.02. DS is columns left of 360 - there is no linear slope. There are plenty of features on the DS above, but none 'slope away linearly from the BS'.
A straight line in a lin-log plot corresponds to an exponential term. The difference between two Gaussians of different width is probably another Gaussian. Are we learning that the real PSF has a Gaussian term in it that varies between filters? Since V-V images did not show this behaviour the Gaussian is not manifested by the inevitable slight image alignment problems. Our model PSF is an empirical core with power-law extensions - and the above experiments show that such PSFs do not yield linear-slope differences.
Perhaps we could study the real PSF by studying difference images in a thorough way? Student project
From flux to AlbedoPosted by Daddy-o Mar 21, 2013 10:49AM
In posts below we have discussed how to best investigate colour differences. Here
we saw that sky brightness and exposure time problems can be detected.
Using selected good images in B and V we found the pairs that also were close in time, and generated B-V images. We noted (discussed here
) that B-V on the BS is not always near the published value of 0.92, even in images selected for not having (obvious) exposure time problems. We wonder if the value 0.92 is more of a classical photometer value? That the colour of the whole BS on average is 0.92? Perhaps - but we also wonder if the reflectivity of the Moon has a phase dependence so that the B-V colour, even if a BS average, is lunar phase dependent?
Here, we choose to bring the B-V value of the BS in our selected images to 0.92 in order to study which values we get for B-V on the DS. This in an attempt to see if we can discuss what the B-V color of the DS, and therefore of Earth is.
For a range in lunar phases equivalent to illuminated fraction from 35% to 50% we plot the B-V values of a slice across the lunar disc, through the centre and 40 rows wide. We average over the rows:
We have aligned the images by the deep cut, which corresponds to the BS/DS border - the terminator. On the right of this we have the BS and on the left the DS.
We see that the BS is level. We have offset the image values so that the BSs are near 0.92 (by eye). We see that the DS has a slope. We see some level differences in these slopes but the slopes themselves are fairly similar. For one of the profiles the B-V reaches as low as 0.5ish, but there is still a slope.
On the basis of that I think we ought to say that "B-V for earthshine is at or below 0.5 in absolute terms". Better may be to say that "the DS B-V is 0.42 below the BS B-V level."
The B-V os sunlight is 0.656 ± 0.005 (Chris has measured a similar value of 0.64). If we also know that the sunlight on the Moon appears to us to have B-V of 0.92 then we can infer that one reflection from the Moon reddens the sunlight by (0.92-0.656)=0.264. The Sun also shines on Earth (lucky us!) and that light has its colour altered by the Earth - when it strikes the Moon and comes back to us it has B-V of 0.5 or less. Knowing what one reflection off the Moon does to the B-V color we infer that B-V of Earthshine is 0.25 or less. Franklin performed UBV photometry [with a 'diaphragm' of diameter near 1 arc minute - it means he observed areas on the DS, preventing the BS light to enter the photometer, but not BS-scattered light from optical elements before the photometer] on the Earthshine and found that B-V 'for Earth' was 0.17 below B-V for the Sun - this implies that B-V for Earth is 0.47. We have a value (from our lowest value) slightly above that.
1) The above slope is rather straight. The halo itself is there because the halo from B and V have not cancelled. This must be telling us something about the scattered light halos? Probably that it is linear in a lin-log plot - which is what we use using the 'BBSOlog' method. What would Ve1-VE2 images look like? We cannot tell since the halo is not (primarily) due to atmospheric scattering and Mette knows no rule for how scattering in lenses depends on wavelength. We have to try, before we know if Vegetation edge data can be extracted in this way.
2) We also see that we have no images without a B-V gradient across the DS. If we had had an SKE we might hope to achieve 'clean' B-V images. But we don't. So we can't.
3) All is not lost - when we fit images profiles directly we compensate for the halo and can therefore extract actual albedo data that does not depend on the presence of the halo.
From flux to AlbedoPosted by Daddy-o Mar 20, 2013 03:13PM
To check consistency we now look at V minus V images, where the two V images are chosen to be closer in time than 30 minutes but not taken at the same time. We correct for extinction; we convert the raw images (bias subtracted, of course) into instrumental magnitude images by calculating the flux from the nominal exposure times and taking -2.5*log10, and then align them and subtract. We shoul dget images of 0s since the fluxes, once corrected for extinction, should show the same flux - at least on the BS where the Sun is shining. We do this and get a royal mess:In each frame the insert shows a color-contour plot of the V-V image. The graph shows the usual slice across the middle of the image, averaging over 40 rows.
We see in upper left panel a clear offset at the terminator - i.e. the DS has different level but the BS are similar. Upper right shows a failry decent pair of images - the terminator is giving some problems and the DS as well as the BS differences are offset from 0 by a small-ish amount which coul dbe caused by an error in exposure time of some 10% or so (not unlikely). Lower left shows that while the two BSs are at teh same level then the DSs differ violently. The lower right shows a really nice example of two images agreeing.
What is going on? We hand-inspect the above images and see that in the case of the lower left image the sky level is much higher in one of the two images used, although they are observed less than 30 minutes apart. The counts inside mare Crisium are 3 and 11 in the two cases - i.e. a factor of about 4 or a magnitude difference of 1.5 - well outside the plot frame. So, from this we learn that we have a method to detect stable sky conditions! It also tells us that using images from different filters should be done with great care - even small differences in sky conditions will cause the DS to shoot off!
As it is, these images are merely bias subtracted - there is no individual correction for a 'pedestal' due to sky conditions. Luckily we can still use these images for albedo work in that we expressly fit a pedestal term!
What else can we learn? Well, any error in actual exposure time will influence the DS and the BS with the same factor - hence the magnitude differences plotted above will become offsets for both DS and BS - hence, upper right is consistent with 'wrong exposure time' in one of the images. Having assigned one filter and received another will have the same effect as an error in exposure time - if the DS and BS are altered by the same factor. I wonder if 'wrong filter' could camouflage as 'wrong exposure time'? Since DS and BS have different colors I doubt it - but we should investigate this.
So, three things learned:
1) Wrong exposure time will lift or depress DS and BS by same amount.
2) More sky brightness in one image than in the other - affects DS only.
3) Image pairs like the ones used for lower right panel, above, are probably both OK.
From flux to AlbedoPosted by Daddy-o Mar 20, 2013 11:09AM
I have located all 'good images' in B and V. That is, all B and V images, made from stacks of 100 images, that are on Chris' list of 'good images'. I have furthermore identified all pairs of B and V images that are taken less than 1 hour apart. For all these pairs I calculate a B-V image, by using Chris' calibration of flux against known standard stars. I allow for an extinction correction based on kB=0.15 and kV=0.10. I get airmasses from the Julian date of the image, and IDL software. I plot a colour contour plot as well as a 'slice' across each image. The slice is made up of an average of 40 rows centred around the row that goes through the centre of the B-V image. A total of about 33 image pairs have been plotted in this way. Here is a (large) pdf file containing the plots.
There are several strange things to see. Generally we get the profiles shown here
. That is - the BS is somewhat flat, while the DS slopes towards the DS sky. Oddities include images where the DS is as flat as the BS - just at another level. Is that 'extremely good nights'? I think not
. In some, the DS is much higher than the BS - that could be images in which the exposure time is incorrect due to shutter problems. Or filter-wheel problems! In many the BS is flat, but not near the 0.92 value we expect based on publications. For small differences with this 'canonical value' I think we could be talking about exposure time uncertainties. If we are at B-V=1.0 instead of at 0.92 we could have an 8-10% error in the exposure time. The times are short and it does not seem unlikely we have that big a problem. Sigh.
I will suggest that we 'correct the BS value' to 0.92 by simple shifting, and then consider how our DS B-V values look. With some 'deselection' of some obviously bad images we can perhaps arrive at a base set of good B-V images.
I think the result will be that few show any leveling off in the 'slope of B.V wrt distance to BS'.
We have few images at small lunar phase (i.e. small illuminated fraction). Chris has analyzed one good pair of B and V images at small phase, but it is not in my pipeline of good images - the automatically detected centre coordinates and radius are way off (because the sicle is so small that automatic methods do not work). Our new student Johanne is inspecting these images by hand and will adjust radius and centre coordinates and then we should have a few more images to play with at small phase. These may show us if we can get far enough away from the BS that the DS slope in B-V levels off and shows the true earthshine B-V color.
From flux to AlbedoPosted by Daddy-o Mar 18, 2013 03:47PM[Note added later: we have updated this posting using extinction corrections for B and V. kB=0.15 and kV=0.1 were used, and airmass=1.67. JD was near 2456016.82]
In preparing for the EGU 2013 conference, where we will have a poster on color results from the earthshine project, I want to show a 'B-V' image of the Moon - since we can, and may be the first to publish such a thing.
Using Chris' calibration of our filters against standard stars in M41 and NGC6633 I can reduce observed images in any filter to 'magnitude images' in the same filters. By taking the difference between the B and V image I thus arrive at a 'B-V image'. Here is a slice across the middle of that image:Second image added later: We now correct fore extinction:
Here, the BS is to the right of the 'cut' in the disk, and the DS is to the left. It appears that the BS is higher than the published value for 'moonshine' of 0.92 [see e.g. Allen, "Astrophysical Quantities", 4th ed, Table 12.14]. Added later: Even after correcting for extinction the B-V is still higher than Allen's value - we now get B-V near 0.95-1.0. Perhaps now, not so much larger than that value?
I am not quite sure about the DS. There is probably a slope because the halo in B and V are not identical and thus do not cancel. But as the BS is approached the color of the halo does approach the BS value, since that is where the light comes from. So I understand that part.
At the extreme left of the DS there is least influence from halo. The value is a bit up and down, but appears to be lower than the BS by up to 0.3. If B-V is a smaller number in one place compared to another it means that B is smaller than V in the first place, relative to the second - i.e. on the DS B is relatively smaller there than on the BS - since these are magnitudes it means we have shown that earthshine is blue - the Earth is indeed 'a blue marble'!
Here is the B-V image itself, with cleverly chosen colours to reinforce our message: Sky is masked out, BS is to the right, DS to the left.
I'll try to redo this for a lunar phase with less halo.
Data reduction issuesPosted by Daddy-o Mar 12, 2013 04:11PM
We have been following the mystery of the VE2 filter. Ana Ulla asked a good question: "Do you see the same problems in images not of the Moon?", so I checked that, using our copious collection of Altair images. I fit a Moffat profile to each well-exposed image of Altair and extracted the sky level. I plot it here as a function of which filter (0=B, 1=V, 2=VE1, 3=VE2 and 4=IRCUT) and as a function of time, for one night (JD 2455845):
In the top panel we see that VE2 (the brownish points) have two levels, and in the lower plot we see that the VE2 level switches from high to low after some time. (Bottom panel has fractional day as x-axis). We also see that the black points (B) increase their level (slightly) at the same point (0.87) where VE2 drops to the low level. Blue points (VE1) have some history like that too. V (red) is quite stable. Green is IRCUT.
If the excess in VE2 (and others) was due to a source of light, then we would hardly see a down-shift in the level. Ana's idea was that thin cirrus could enhance the nIR band of VE2 (quite like the Johnson I band) - but that does not explain why B rises, unless that is a particular behaviour of cirrus clouds, which I doubt. I do not think anything above is consistent with a light source. I think this is what we have had to conclude from the other results presented in this blog - e.g. the inverse exposure time dependence and the inverse airmass dependence.
I am guessing when I say that I think it is something to do with electronics. The camera knows which filter is being used! (Not
From flux to AlbedoPosted by Daddy-o Mar 11, 2013 09:53AM
We have already shown a strange behaviour of the VE2 filter - or what appears to be a problem related to the use of the VE2 filter. The post is here
Now we are able to report something even stranger (OK, just
as strange then). We appear to have a relationship between the 'VE2 pedestal' and the airmass of the observation:
In the upper frame we see the old VE2 pedestal vs exposure time plot. In the lower frame we see that the pedestal is also a function of the airmass! If the pedestal was proportional to airmass we could, perhaps, understand the phenomenon as something showing a flux proportional to the amount of atmosphere (I am thinking of some sort of airglow here) - but NO, the pedestal height is smaller
the larger the airmass is!
Is this some electronic problem related to how the telescope is oriented? Then why the inverse proportionality on exposure time too? Why not the other filters?
going on here??
Data reduction issuesPosted by Peter Thejll Mar 08, 2013 09:25PM
During reduction of our data it has become evident that the VE2 filter gives us certain problems. A new problem is that VE2 exposures tend to have a 'sky component' that is much larger than in the B,V,VE1 or IRCUT filters. Here is a plot of the relationship between exposure time and the 'pedestal' - i.e. the offset that has to be subtracted from the VE2 image to bring the sky level down near to 0 (in bias-subtracted images, of course).
It does seem like the pedestal height is a decreasing function of the exposure time. Why is it any
function of exposure time, when the other filters do not show this behaviour?
If there was a source of light in he image frame, then its level should INcrease
with exposure time, surely? Here is the plot for the VE1 pedestal:
The level is much smaller than in the VE2 case - and it does not appreciably depend on exposure time.
What is going on?
From flux to AlbedoPosted by Daddy-o Feb 28, 2013 10:08AM
In this entry: http://iloapp.thejll.com/blog/earthshine?Home&post=304 we investigated the effects of using a lunar albedo map based on scaling the Clementine map to the older WIldey map vs. scaling the Clementine map so that lunar mare and highlands matched what is published in the literature. [Note: the point being that the 'Clementine map' we have available is just a picture file - jpeg! - so that pixel values have to be scaled to albedo values somehow.]
We were doing this fitting in the 'new way' which is to fit 'profiles' starting on the sky on the DS side and extending onto the DS itself, modelling the contribution from the scattered BS light. In doing this we saw that improved fits could be obtained if the lunar map albedo was 'stretched' so that dark and bright areas better matched our observations. We did this stretching by eye and were able to improve the formal fits.
We have now compared the fits on 535 images done using the 'Clementine scaled to Wildey' map and the 'Clementine stretched by eye' maps.
We have found that the RMSE (that is, the square root of the sum of the squared residuals [i.e. observed profile minus best fitted model profile]) is improved using the scale by eye' map - in two ways: a subset of images were poorly fit using the other map; they are now much better fit, and the mean RMSE of the finally selected images is lower.
We selected 'good fits' on the basis of alfa (the PSF-width parameter) having to be in a narrow range and that the relative uncertainty on the fitted albedo should be below a certain limit.
Mean log10(RMSE) is now near -1.1 in units of counts/pixel. We fit a profile that is 150 pixel columns long - 50 pixels on the sky and 100 on the DS.
Mean relative fit-uncertainty on the albedo is near 1.5% when using 'counting statistics errors' on the observation.
We note that the VE2 images more frequently have a larger 'pedestal' or sky offset after bias removal than the other filters. While most filters have an offset of near 0.2 counts (+/-0.5 or thereabouts) the VE2 offset is more often near 4 or 5 (+/- 1-2 counts). What is the cause of this? The bias frames surrounding the VE2 exposures have been spot-checked and seem OK - bias is near 400. A few, otherwise perfectly all right, VE2 images have a sky offset of 10-40 counts! Observations from the same night in other filters show nothing like this - some sort of nIR 'fog'? Does the sky emit nIR light?
Does this have a conseqeunce for the 'tunnel selection' of VE2 images, done by Chris? [A data-selection method designed to take into account shutter and filter-wheel problems by requiring that image total fluxes follow a known phase-curve.]
While the 'new fitting method using stretched lunar albedo maps' formally works best of all methods we have seen so far, it is very empirical. Later we may be able to use the method to produce a 'best fitting lunar albedo map' that improves on the original Clementine map. We may then also be able to compare to what the LRO people (and LLAMAS) are finding.
From flux to AlbedoPosted by Daddy-o Feb 25, 2013 08:49AM
After the scattered light has been removed we still have the task of converting the observations to a terrestrial albedo. This is done, by us and the BBSO, with the use of a model for how the Moon reflects light. This involves assuming a lunar albedo.
The BBSO has a set of lunar-eclipse observations that they use to find the DS/BS patch albedo ratio - we use the Clementine map. That map is available to us only as an image file. We scaled that image so that the lunar mare and highland albedos correspond to what the literature states.
An alternative would be to scale it to match what lunar albedo maps show. The only digital lunar albedo map we have is the 1970s Wildey map, which Tom Stone gave us. In this blog: http://iloapp.thejll.com/blog/earthshine?Home&post=253
we performed a regression-based scaling of all the pixel values in the Clementine map to corresponding (or interpolated) pixel values in the Wildey map, using the two maps' coordinate systems.
We therefore have the possibility of fitting our observations to the lunar map of Clementine scaled to Wildey (let us call that the "Wildey map" from now on) or the Clementine map ("Clementine map" - but note that both are Clementine-based!) scaled to the highlands and mares. This choice has been investigated.
We find that the contrast in the Wildey map is lower than in the Clementine map, which affects the quality of fit.Same lunar-edge profile (white line) fitted with two different models (red line) - one uses the Clementine lunar albedo map, scaled to match lunar highlands and mare; the other is the Clementine map scaled, using every pixel, to fit the Wildey lunar albedo map. Notice the smaller contrast between highs and lows in the upper image (Clementine scaled to Wildey) compared to the lower (Clementine scaled to highlands and mare).
There is evidently a difference in the quality of the fits - neither being particularly good, failing to match the observed highs and lows. The conseqeunces for the albedo fitted is at the 3% level.
The mean difference in the terrestrial albedo determined using the two maps:
16 albedo values determined using both Clementine-scaled-to-Wildey (vertical axis, in top plot) and Clementine alone (h. axis). We see that the two albedos are near the diagonal - with only a few outliers. The histogram shows the absolute difference between the two albedo determinations - it is about 1% of the albedo value.
As long as the large spread in albedo values (from 0.23 to 0.42) is real and not due to some data-treatment bias, we have an effect from the choice of lunar albedo map that is at the level of the errors from pixel statistics (shown elsewhere in this blog).
We would like the known biases to be less than the effects due to scatter, so perhaps some work on the lunar albedo is in order? We can look at the mean values of the two maps - there was no requirement that mean albedo be conserved, and we can look at various ways of stretching the contrast, while maintaining mean value - this is a nice piece of work for a student project.
Performing a hand-adjustment of the contrast scaling, while maintaining the original Clementine map mean, we can refit the relevant profile. We get this:
Smaller RMSE, small but important change in ftted albedo.
From flux to AlbedoPosted by Peter Thejll Feb 23, 2013 06:53AM
The best data will be those that have almost no interference from the halo at all. Those exposures, if they can be identified, will have the least scattered light on the DS and will therefore be the easiest to clean up. Here we try to identify those exposures. We simulate the effect of the halo on ideal images and measure the amount of flux added by the halo - before any attempts to remove it. We loop over relevant values of lunar phase and 'alfa', the PSF-width parameter. We calculate the change in flux for a region near the DS rim and express the change in percent. The contour plot shows contours at 1,2,4 etc percent change. Overplotted are red data points showing locations of real exposures.
We see that the least polluted images are found at large phase (near New Moon) and for narrow PSFs (large alfa values). This is not surprising! We see that we have several exposures taken under conditions that allow just 1,2,4 etc percent change in DS intensity.
If these images are the least polluted by BS-scattered light, we should have the most success when applying our scattered-light removal techniques to these. If they have a 10% effect due to scattered light, and we can remove 99% of that we have reduced the error induced by the BS halo to the 0.1% level.
Showcase images and animationsPosted by Daddy-o Feb 21, 2013 09:17AM
During processing a strange signal was found in a frame combined from 100 images. It turns out that in one image of this stack we have what appears to be a meteor or a satellite flash - or something:
The image is from Jan 17 2012: (UTC 2012-01-17T13:28:48). The exposure time was 0.009 seconds! The trail is about 3/4 of the lunar diameter in length - i.e. about 22 arc minutes. The orientation is such that it is travelling almost Due North (or South!).
What is it? Well - it is clearly between us and the Moon! If it is a meteor its height would be something like 50-100 km. The speed would then be 35 - 70 km/s. A satellite in low earth orbit has speed 8 km/s. Since the image is taken from Hawaii at UTC 13:28 it is near midnight on Hawaii - i.e. the Sun is behind the Earth and unlikely to be illuminating a LEO satellite.
An airplane flies at 800 km/hr at altitude 10 km, so the distance covered in 0.009 s is 2 m which would subtend an angle of 1 arc minute. This is no airplane - or it is much closer, in which case we should see details of the plane.
As far as I can tell there is no pronounced peak in meteorite activity in January.
From flux to AlbedoPosted by Peter Thejll Feb 17, 2013 08:21AM
We select 6 nights and extract albedos in 4 bands:
Get better pdf file here:
Here are albedos determined for 6 nights. We see 4 filter bands (VE2 is still being processed). Each color represents one night. There are 3 determinations of albedo from each image - hence points come in columns of three.
1) Albedos can differ in level from night to night - e.g. blue and red in panel 1.
2) Albedos can also be almost constant during a night - e.g. purple and green points in panel 1 vs red or blue.
3) Some nights have large scatter - e.g. green vs blue
4) B albedos are higher than V and others
5) VE1 and IRCUT albedos are very similar - the filters are also very similar, so this is good for reproducibility.
6) While some albedos evolve smoothly on a given night IRCUT shows a 'dip' - e.g. blue crosses in B vs blue points in IRCUT
Questions of interest:
a) Why are some nights noisy? E.g. the green points. Information on airmass, alfa and bias frame statistics, and RON, are available.
b) Can the nightly evolution (i.e. slope) of the points be related to anything happening on Earth?
c) Can the change in slope in IRCUT be related to anything special?
Data reduction issuesPosted by Daddy-o Feb 12, 2013 11:41AM
In several posts we have considered the behaviour of variance-over-mean images (VOM). In the ideal world these should be 1 because the noise (once RON is subtracted) is Poisson-distributed. We have seen how this is hardly the case in observed images.
We now consider a test using ideal models with and without imposed image drifts. We generate 100 synthetic images with Poisson noise, and RON as well as a bias.
Below, each set of three panels show a cut across one image in the synthetic stack, then a slice of the VOM image before image alignment and then the same slice after alignment. On the left are images that were not drifted while to the right images were allowed to 'jiggle around' by a few pixels.
We see that VOM is 1 on the BS in un-jiggled images, but that DS and sky values fall below 1.
We see a HUGE effect on images that were allowed to drift.
Some of it we understand. Before alignment
, VOM rises on structured areas of the drifted images because surface albedo variations are being introduced for a given pixel along the stack direction. The effect on the DS and sky is much less - perhaps because the Poisson noise is so large comapred to the signal variations. After alignment
, VOM falls to slightly below 1 on the BS, except near the edge. On the DS and the sky, though, a large lowering is seen. So far it is not understood how this comes about.
Any strange effects seen in observed images will be all the larger since the images do not just drift but also 'shimmy' because of atmospheric turbulence.
The effects of aligning images to sub-pixel shifts is part of the above.
Let us learn from this that a noise model probably is hopeless to build in the presence of image shifts - despite realignment - and that sub-pixel interpolation is not a welcome added bother. We could just omit single images with the most drift and use a straight average of the remaining stack. In real images we have the option to not use stacks that have a l ot of drift - but we do not know the extent of the 'shimmy' for the remainder.
These realizations above have the most impact on our ability to interpret results that discuss the effect of alignment - alignment reduces some problems but probably adds some others.
Error budgetPosted by Peter Thejll Feb 10, 2013 01:01PM
In this post: http://iloapp.thejll.com/blog/earthshine?Home&post=297
we investigated the effects of alignment on the noise. We measured noise in squares on the final image and therefore had some 'cross-talk' from the variations due to surface structure.
Now, we consider the variance-over-mean along the stack direction - that is, for each image pixel we calculate variance and the mean and look at the ratio of these, which we call 'VOM'. In the perfect world, this image should be 1.
We look at unaligned images, and then at aligned images. The 'superbias' was subtracted (but not scaled to the adjacent dark-frame average). RON of 8.3 electrons was subtracted before VOM was calculated as (SD-RON)^2/mean. Gain used was 3.8 e/ADU. We plot VOM in a slice across the final images:
Top is VOM in the unaligned image, and below is VOM in the aligned image. A lot can be mentioned: First, the surface is not constant, obviously. Second, the effect of alignment is not just a uniform lowering of the VOM that we expect (same mean, less cross-talk between pixels).
In the top image we have VOM near 0.1 (dotted line) on the DS and most of the sky. On the BS the VOM is near 10 apart from the peaks that occur at the intersection with the lunar disc edge. There variance rises because of jitter between images and the mixing of disc and sky pixels.
In the aligned image VOM is near 2 or 3 on the BS disc, higher at the peaks (so there is still an edge-effect, but less). On the DS and the sky a spatial structure has been formed, slanting inwards towards the BS.
What is going on? Effects of pixel-subshifting? What does the interpolation do? Why is it spatially dependent? Strange flux-nonconservation?
The sub-pixel shifting used is based on the IDL 'interpolate' function. In the manual for this function it is suggested to allow for cubic interpolation by setting a keyword "cubic=-0.5". I did this and recalculated the above. The effect was slight.
A test was performed to see what variance is induced by the interpolation. By shifting an image by a non-integer amount of pixels in x and y and then shifting that image back, and calculating the difference, we see that using INTERPOLATE without 'cubic' induces 2-3 times more residual S.D. than does use of INTERPOLATE with cubic=-0.5. The interpolations lost variance compared to the original image. With cubic interpolation the loss of standard deviation relative to the mean is at the 1% level - quite a lot actually. [Note: conservation of flux not the same as conservation of variance]
Could it be some effect of bias-subtraction? VOM has dropped most in the halo near the BS. Why?
Control SoftwarePosted by Peter Thejll Feb 09, 2013 03:19PM
In a potential upgrade of what we currently have, we might consider going to linux and freeware, for our command system.
Some devices can be operated via the indiserver concept. Devices for which drivers have been written can be commanded from essentially the linux command line. This means that scripting systems are easier to set up - for us at least.
I have been able to control our SXV-H9 CCD camera using this system. Here is a link to how it is done:
There exist indiserver drivers for gphoto2 - that means that Canon DSLR cameras, for instance, can be operated in this way.
A TrueTech filterwheel is being tested. LX200 telescopes can be controlled - i.e. mounts such as ours.
Data reduction issuesPosted by Daddy-o Feb 08, 2013 12:37PM
It would be a Good Thing if we could explain observed images in terms of a 'noise model' - that is, a model that explains why the signal and the noise bears a certain relationship.
We consider here a stack of 100 raw images taken in rapid succession. The difference between adjacent images should only contain noise in that any objects shown in both images should subtract. The difference between two series drawn from the same poisson distribution but independent has a variance that is twice the variance in either of the series subtracted, so we apply a factor of 1/2 to the difference image to get the variance image.
We subdivide such a variance image into smaller and smaller squares, and in each square we calculate the variance. We generate an image of the ratio between the variance image an dthe mean original image, also subdivided In the case where everything is Poissonian this image should be a surface with value 1. In reality there will be noise in this surface - and there will be strcutures seen wherever imperfect obejct subtraction took place. Here is the result:In the three rows we results from different subdivisions - into 8x8, 4x4 and 2x2 squares. In the leftmost column is a histogram of the values of the image and to the right is plot of the profile across the middle of the image.
We see, to the right, that variance over mean (VOM) is not 1 everywhere. We are evidently picking up a good deal of variance near lunar disc edges [The image corresponding to the above is a quarter Moon with the BS to the right and DS to the left, situated in the image field center]. We see that the sky manages to have VOM near 1 and that parts of the DS does this, but that most of the BS appears to have VOM>1. Even ignoring the peaks that are due to edges we see a value for VOM near 2 or more [also seen in the rest of the 100-image sequence].
Since we used raw images we have to subtract a bias level. We subtracted 390 from all raw image pixels. We applied the ADU factor of 3.8 e/ADU to all bias-subtracted image values. We then calculated the variance and subtracted the estimated RON (estimated elsewhere at near sigma_RON=2.18 counts, or variance_RON=4.75 counts²; consistent with ANDOR camera manual "8.3 electrons").
The subtracted bias is a little small - the observed mean bias value is nearer 397, but if we use that value we get strange effects in the images - only a relatively low value of the bias mean gives a 'flat profile' for the sky in a slice across the image. This is one poorly understood observation.
We also do not understand why the VOM is nicely near 1 on the DS while it fails to clearly be near 1 on the BS - both areas of the Moon have spatial structure which is bound to contribute to the variance in the difference image during slight misalignments.
That is the second poorly understood thing.
Progress towards a 'noise model' is therefore underway, but there is some distance to go still.
How would non-linearity in the CCD influence the expectations we have from Poisson statistics?
Data reduction issuesPosted by Daddy-o Feb 07, 2013 10:42AM
We are using stacks of 100 images. These are acquired by the CCD camera in rapid succession - 100 images can be obtained in less than a minute. During that time there may be small motions of the telescope, and the turbulence in the air, as well as the slight change in refraction due to airmass changes may cause image drifts in the stack.
In one approach we ignored the possible drift and just averaged the 100 images. In the other approach we use image alignment techniques to iteratively improve the alignment of the stack images: First we calculate the regular average, then we align all stack images against that average image, then we calculate a new average image on the basis of the alignments and re-align all images against this average and so on. This procedure turns out to converge, and after 3 iterations we can stop and save the last average image.
We would like to know the effect of doing this on result quality. We therefore generated two sets of averaged images - the first using the simple first kind above, and the second the iterative method.
We estimate 'errors' in the bootstrapping way. That is, we extract MHM averages (mean-half-median, as explained elsewhere) of the DS intensity on raw and cleaned-up image patches and also estimate the statistical error on these values by bootstrapping the pixels inside the patch, with replacement. This bootstrap procedure gives us a histogram og MHM values and the width of this distribution is a measure of the 'error on the mean'. We express this error as a percentage of the mean itself, for RAW images and images cleaned with the BBSO-lin, BBSO-log and EFM methods.
we now compare the results to see the 'effect' of performing alignment of stack images:
Table showing errors (in percent of the mean). Lines labelled 1: and 2: show results for 'one-alfa PSFs' with and without alignment. The third line shows the effect of using a 'two-alfa PSF' and alignment.
We see that there has been a large reduction in errors by using alignment. Raw images improved by 20%, EFM images by 17%, BBSO-lin by 16% and BBSO-log by 14%.
The effect of using a two-alfa PSF on aligned images is small - indeed, all images except EFM experience a small increase in the error (probably not significant).
The effect of alignment on single-alfa PSFs is not investigated.
We conclude that alignment is a beneficial operation. We note that EFM has the largest errors but other arguments imply EFM is the better method to use - this is related to the stronger phase-dependence seen in non-EFM images, and is discussed elsewhere.
Data reduction issuesPosted by Chris Flynn Feb 06, 2013 11:10PM
Our standard practice to date has been to take 100 exposures of the moon, each with an exposure time of 10 to 40 milliseconds, depending on the brightness of the moon. The exposure time is set so that the bright side attains about 50,000 counts maximum, in order to get good signal but not risk overexposure (saturation of the CCD wells).
We have looked at the differences between successive images in these 100 image stacks. A good night with a fullish moon was chosen (JD2456106)
and the relative difference between the first 50 consecutive pairs of images in the stack computed, and made into a movie -- herewww.astro.utu.fi/~cflynn/reldiffs.ogv
(this can be viewed with mplayer in linux)
An example of nine consecutive frames is shown below :
During the sequence of 50 exposures in the movie, the moon only moves about 1 pixel down and 1 pixel to the left, so drift in the position of the moon from one frame to the next is practically negligible.
White and black on the grey scale run from a -10% to a 10% difference with the previous image. Note that the two compared frames are normalised to the same total flux (this is typically a correction of ~1%).
We see plenty of structure in the difference image, which we interpret as being due to air cells -- turbulence -- above the telescope. The telescope size (3cm) and short exposure times (20 ms) mean that scintillation is very significant (because of the turbulence). In tests on bright stars with our telescope and 10 ms exposures, we were getting scintillation fluctuations in the fluxes of order 50% -- very large -- as one would expect (Dravins et al 1997a) (paper attached below).
Some of the structures look greatly elongated, and remind us of the "flying shadows" discussed by Dravins et al 1998 (paper attached below). To quote them : "Atmospheric turbulence causes “flying shadows” on the ground, and intensity fluctuations occur both because this pattern is carried by winds and is intrinsically changing".
Analyse sequences of images in 100 image stacks, and make an estimate of the amount of turbulence present following Dravins et al's work -- for correlation with our fitted parameter alpha of the PSF -- to see if the amount of turbulence in the frames has an effect on the PSF's long power-law tail.
Dravins et al 1998:
Dravins et al 1997a (the scintillation figure is on p 186):
Dravins et al 1997b:
Data reduction issuesPosted by Peter Thejll Jan 26, 2013 05:29PM
In this post: http://iloapp.thejll.com/blog/earthshine?Home&post=293 I proposed to fit more elaborate PSFs than we have been using.
I present here the results of fitting a 2-parameter PSF. The previous one was characterized by an exponent alfa, and had the form
The new one allows two exponents - one regulating the slope of the PSF up to a cutioff point in terms of radius, and the other parameter taking over outside that point. Thus we have 2 new parameters to fit - exponent and cutoff point.
Using such a PSF and evaluating success on the parts of the image designated by the mask - and using a mask consisting of sky on both sides of the Moon, but no polar areas - i.e. a 'belt across the image' - we arrive at images with slight changes in the estimated halo on the DS of the Moon and larger changes in the area on teh BS-sky.
This plot summarizes what was found:The top panel shows absolute value of residuals (obs-EFM-model) in a slice across an image after the halo has been removed (along with most of the BS, as is always the case in the EFM method). The PSF used was the new tow-parameter PSF.
The middle panel shows the result when the usual one-parameter PSF has been used.
The last panel shows the difference between the two panels, expressed as a percentage of the first one. The illegible y-axis label for the bottom panel shows 10^-2 at the bottom, next label is at 10^0
Note that absolute
values of residuals are plotted above.
We see the DS to the left, the almost-erased BS to the right and the sky on either sides.
Clearly, the BS sky has been removed much better in the first image - i.e. with the two-parameter PSF. On the DS we see changes in the range from 1 to a few percent on the sky-ward part, rising to tens of percent near the BS.
This implies that a signifcantly different amount of scattered light has been removed using the two PSFs - but which is the better result? Judging by the RMSE per pixel on the mask in the two cases there has been a significant improvement in going from one to two parameter PSFs. The RMSE per pixel in the one-parameter PSF case is about 5 [counts/pixel], while the RMSE per pixel is 0.23 [counts/pixel] in the two-parameter case. Most of this change has evidently taken place in the BS part of the sky.
We need to know if there has been an improvement on the part of the image that matters - the DS!
The best fitted parameters were quite alike - near alfa=1.7. The best-fit cutoff point was near 30 pixels.
Exploring the PSFPosted by Peter Thejll Jan 26, 2013 10:45AM
It seems that success for us will be linked to our ability to correctly remove scattered light from the BS. This hinges on our understanding of the scattering model - in essence, the PSF.
Currently we use a PSF that is empirical. The core is made up of values from a table generated from observations of bright stars and Jupiter. To that we link an extension, also empirical, that is based on what the far wings of the lunar halo looks like. This PSF is then 'exponentiated' during fitting to actual images in the EFM method.
We noted during several posts below that the values for the exponent varied little and only now and then seemed linked to the extinction. This could be an indication that most of the time (during good nights) we are limited by something fixed - such as the optics - rather than atmospheric conditions. On the other hand it could mean that the PSF is not very accurate in its basic from and that the fitting method gives up at some stage, leaving us with an exponent that is somewhat random, and therefore not linked to the atmospheric conditions.
We should also recall what happens during application of the EFM method: We have tested various forms of sky masks for this method - some that allowed fitting emphasis on both the DS and BS sky, and others that emphasized only one side. Common for the ones that focused on either just the DS sky or the BS sky was that the halo shape on the other side was not very good.
Might these things be indicating that a better PSF should be generated or a better way found to apply the fits?
I'd like to suggest the following: Perhaps the PSF has a form like
instead of the present
PSF ~1/R^const_alfa ?
I would like to try to use a piece-wise constant alfa so that the PSF is separated into radial zones, each of which has its own alfa, found by fitting.
More to be added ...
Optical designPosted by Peter Thejll Jan 26, 2013 10:06AM
In this post: http://iloapp.thejll.com/blog/earthshine?Home&post=289
the importance of the SKE for scattered light was discussed. The images shown, though, were not shown fairly - with intensities scaled to comparable levels. I therefore extracted a line across the BBSO image and a line across our image, at right angles to the SKEs, rescaled the intensities, aligned the plots and get this:The black curve is from our image (whichis a sum of 10 well-exposed images). The red curve shows the cut across a single BBSO image.The BBSO image has only the DS peeking our behind the SKE, while our image has the BS in full view.
We see entirely comparable 'halos'! The BBSO image ha a more pronounced sloping 'tail' onto the black side of the SKE than we do, and more noise. If that sloping tail is 'halo' we had a better system than the BBSO!
What does the above mean? It does NOT mean that we have less 'halo problems' than BBSO does - because the BBSO expose their DS so that the halo from the BS is not allowed to be formed. Yes there is a similar halo from the DS on their images as there is from the BS on our images - but the halo from our BS is very much stronger than their DS halo.
When the BS halo is small - i.e. near New Moon - we have minimal effect of the BS halo.
From flux to AlbedoPosted by Peter Thejll Jan 25, 2013 04:02PM
We have now reduced all the good data, and have arrived at the intrinsic properties of each image. We extract now the ratio of the DS-patch to the total flux and divide this by the similar ratio extracted from the synthetic model. This ratio of ratios is the same as the 'Lambertian sphere terrestrial albedo' and is the quantity published by e.g. Goode et al.
We select data for airmasses less than 2 and for statistical error less than 1.5%. This error is that due to mainly pixel-selection inside the 'patches' on the lunar surface (one near Grimaldi, one inside Crisium). Monte Carlo bootstrap sampling of the pixels inside a lon-lat box were resampled, with replacement, and the standard deviation of the consecutive means calculated. Errors due to image alignment, improper scattered-light removal, synthetic model problems and so on, are not included.
Here is the plot of extracted albedo against lunar phase for the VE1 filter (the filter with least scatter: IRCUT is almost identical to this plot): [scroll down for discussion]The colours designate the method used for scattered light removal: red is the EFM method, orange or yellow is the 'linear BBSO method', and green are the raw data. Crosses and diamonds indicate the sign of the lunar phase - crosses are for positive phase.
Error bars (error due to counting noise, not image alignment etc) have been plotted over the symbols, but are all smaller than the symbols themselves.
What do we see? There is scatter and there is dependence on lunar phase.
First, let us discuss the lunar phase dependency: For small phases we are essentially getting closer to Full Moon and the scattered light is more and more of a problem. The Raw data are doubled by phase 80 degrees, showing the importance of removing the scattered light. BBSO-linear is better than no removal all while EFM is better than BBSO-linear. There is also another lunar phase dependency - notice how the albedo rises for large lunar phases. Since the halo is smallest here this is not an effect of scattered light! The albedo is a composite quantity, consisting of quantities measured in observed and modelled images. The scattered light influences mainly the model, and mainly at small lunar phases. The reflectance model soley influences the synthetic lunar image, of course - I think we are seeing the influence of the (inadequate) reflectance model at large lunar phases - the albedo rises artificially due to this.
The there is the scatter. We see 'daily progressions' for some connected lines of data points. For instance, by 90 and 95 degrees phase. These points represent our candidates for geophysics! I think the 'progression' along these lines are due to different parts of Earth )oceans, clouds, continents) rotating into view for these observing sequences.
Below is the same plot, for the B filter. Apart from more scatter we see that the EFM seems able to remove most effects of scattered light, compared to the BBSO-linear method. A stronger 'upturn' is possibly seen at large phase.
Data reduction issuesPosted by Peter Thejll Jan 23, 2013 03:24PM
Chris thoroughly investigated the quality of all data and arrived ta a list of the 534 best images: http://iloapp.thejll.com/blog/earthshine?Home&post=277
I have collected all those images in 'cubes' consisting of raw observation, EFM- and BBSO-cleaned images, images for incidence and emergence angles and the ideal synthetic image and lunar longitude and latitude images - each 'cube' contains centred images for a given 'stack' of 100 observations. We thus have 534 'good coadded' images with all secondary data that belongs to that moment. Each cube is about 16 Mbyte uncompress,a nd slims down to 5-7 Mbyte upon being bzip2-ed.
I inspected the correlation of these observed stacked images with their synthetic models and found that 8 of them are poorly correlated with their synthetic models. I inspected each case and found that the problem was either focus (the observed image is blurred) or a slight alignment problem.
Omitting those images we arrive at this list of 525 best images:
These are all available as 'cubes', upon request.
In the future we might wish to only work with these images, since alignment and image-reversal issues have been dealt with (the observed images swap East and West upon the meridian flip).
We can set up a Mark I dataset, to be updated later as we find issues or solve the remaning (small) alignment issues.
Post-Obs scattered-light rem.Posted by Daddy-o Jan 22, 2013 01:39PM
We have some images of the SKE inserted in the ray path of our telescope, during Moon observations. We also have some images of the same from the BBSO telescope, kindly lent to us by Phil Goode.BBSO images om two separate periods about a week apart. Top row shows histogram equalized imges of the BS (left and the DS (right) with SKE superimposed over the BS. Lower row shows the same - BS short exposure on left and DS long exposure with SKE on right.Histogram-equalized image of the Moon with SKE inserted from our own telescope. SKE runs from upper left to lower right and is opaque above that line.
Our image shows a halo around the Moon that stretches behind the SKE - hence we know that halo is formed to a large degree in the secondary optics! On the BBSO images we do not see this as clearly - either because the BBSO telescope is of a different design or because the exposures with SKE (right column of images) covers much more of the BS. Assuming the latter we understand why there is no halo behind the BBSO SKE - the BS was effectively blocked and no bright light entered the secondary optics and could not cause a halo. On our image the SKE is inserted experimentally only, leaving a large part of the BS to shine into the secondary optics.
So, we learn that the halo is not from atmosphere or primary optics alone - apparently a large fraction of it comes from the secondary optics!
We also see the importance of having an SKE. While the BBSO group uses the 'linear BBSO method' to remove scattered light over the DS they have a much smaller problem than us because the halo from the BS is nowhere near as strong as ours is!
We now see how terribly important the SKE is.
From flux to AlbedoPosted by Daddy-o Jan 22, 2013 09:09AM
Following on from post
we now inspect satellite images one week apart in order to understand the natural variability found in satellite images of the same area.Upper panel: average image pixel value for sequences of MTSAT images, 1 hour apart, for almost one day in March 2013 and a week later (red curve). Image pixel values is in arbitrary units but is proportional to pixel brightness. Lower panel: difference between upper panel black and read curves, expressed as a percentage of the mean of ther ed and black curves.
The difference between the two curves is on the order of 10% and varies from 8% to 18% during one day.
This tells us two things: Albedo (or something proportional to albedo) can vary by roughly 10% over a week. Albedo can vary during one day by almost as much.
This is useful information to have when we interpret the earthshine data.
We keep in mind that the smooth variations in the black and red curves in panel one are due to the day/night cycle - not intrinsic albedo variations: but the difference
between the curves and the variability in the difference tells us about albedo variations.
From flux to AlbedoPosted by Daddy-o Jan 21, 2013 09:35AM
We can reduce our observations to a number that is equal to the 'Lambertian albedo of Earth' at the moment of observation. That is, for a sphere behaving like a Lambertian sphere we can find the single-scattering albedo that gives the same earthshine intensity as we observe. This takes phases and all relevant time-dependent distances into account.
In order to understand if the data we get are realistic we wish to compare to satellite images of the Earth. From work published by others [the Bender paper] we are told that the terrestrial albedo varies by many percent from pentad [avg, over 5 days] to pentad.
Many of the best observations we have have 'sunglint coordinates' over South-East Asia. One geostationary satellite hanging over that spot is the Japaneese 'MTSAT'. We may be able to get data from it, but have only ready access to a Meteosat that hangs over the Indian Ocean.
From the Indian Ocean Meteosat we have extracted a series of images for a given day, in order to start to understand what sort of variability we shall expect on time-scales that are shorter than a pentad.
We have access to half-hourly images from the satellite and have 12 hours of data - 25 images. We take the average of each image and plot it:
The black line is the observed mean intensity of the whole-disc image, and the red curve is a 6th order fitted polynomial. The difference between the black and the red curve has a standard deviation of 0.6% of the mean of the black curve.
The large-scale behavior of the black curve is due to the phase - we see half a day pass as seen from the satellite so Earth changes phase from new to full to new again. Variations over and above that would be due to changes in earth's reflectivity. There is a pronounced sunglint in the Arabian Sea near Noon.
Some pixels in the image are saturated.
What do we learn from this?
By removing the fitted curve we learn how much variability there is as a day passes. Some of this would be due to the curve not actually being a 6th order polynomial - so the variability we get from the residuals are an upper limit. Fitting a 7th order polynomial lowers the S.D. to 0.4% of the mean.
It seems that the presence of a sunglint for some of the frames (the sunglint moves on to land - Horn of Africa - in the local afternoon and does not 'glint' in the sea anymore) does not generate a 'spike' of any kind in the average brightness.
We should remember:
That average brightness is not the same as albedo since the brightness depends on albedo times a reflectance function. But most of the reflectance behavior is removed via the polynomial fit, we think.
That cloud-patterns on this scale hardly vary much.
We do not yet fully understand the processing of the satellite images - are they normalized somehow? Were they all taken with the same exposure time?
On this day at least variability in the albedo was much less than 1%.
We should inspect several days of images to see if the average level differs much from day to day.
We should try to get information from MTSAT, and at higher cadence - we can observe the earthshine with minute spacing.
Try to get more technical information about the satellite images.
Relevant papersPosted by Peter Thejll Dec 31, 2012 09:26AM
This is the digital atlas of the Moon produced by Wildey which we scale the Clementine map to.
From flux to AlbedoPosted by Peter Thejll Dec 30, 2012 10:41PM
The Laplacian method has been discussed here
We now extend the method to include an identical analysis step applied to synthetic images generated for the moment of observations. We then take the ratio of the results from observations and the results from the models. This will eliminate the effects due to geometry and reflectance (as long as the reflectance model used for the synthetic images is correct), leaving only the effects of changes in earthshine intensity. So, this is another 'ratio of ratios' result. The ratios involved are, in summary:
(Laplacian signature at DS/ Laplacian signature at BS; in observations) divided by
(Laplacian signature at DS/ Laplacian signature at BS; in models).
We use only the 'good images' identified by Chris in this entry: http://iloapp.thejll.com/blog/earthshine?Home&post=277
by inspecting the dependency of (Laplacian signature at DS/ Laplacian signature at BS; in observations) on phase [attention: this is not yet the promised ratio of ratios - just a ratio! :)], in each of the available filters:
We see the morning and evening branches folded to the same side, for comparison. We see an offset between the branches in each filter, and we see the dependency on lunar phase. We next look at the same ratio but from models. Since we do not have colour-information in our models we redundantly now show plots for each filter as if the models were color-dependent - they are not: but the points available in each filter are different, of course. Recall that a model is generated for each observation:
We see a very similar pattern - dependency on phase (but steeper this time). We also notice that the branches are closer together than in the observations.
What does this mean? The branches are separated, in observations, due to different distributions of light and dark areas on the eastern and western halves of the lunar disc facing Earth. Our Laplacian method samples pixels right at the edge of the lunar disc and has evidently met areas of different albedo. The separation in observations is not reproduced in the models - this can imply that the model albedos are incorrectly distributed (i.e. mares and craters etc in the wrong places) this is somewhat unlikely as we use one of the most detailed lunar albedo maps, from the Clementine mission. However, our model does not sample colour - the map used, and thus the ratio between 'light' and 'dark' - is taken from the 750nm Clementine image [I think; must check!]. This map was stretched to match the older Wildey albedo map [see here: http://iloapp.thejll.com/blog/earthshine?Home&post=253 ] which was made in such a way that the 'filter' the WIldey map corresponds to is a combination [see: http://iloapp.thejll.com/blog/earthshine?Home&post=286 ] of the Johnson B and V filters. We see the most well-reproduced branch spacing in the V band observations, compared to the models. This implies we have some colour-information about the two halves of the Moon - or a tool for how to scale the lunar albedo map when different colours are to be considered.
We next inspect the ratio of the observations and the models - the ratio of ratios:For each filter is shown the Albedo derived - it is the 'ratio of ratios' spoken of above and is identical to the one used in BBSO literature - it is the albedo relative to a Lambertian Earth-sized sphere. The model albedo used was 0.31 so the 'actual' albedo derived is the above times 0.31.
We see that there is a phase-dependence in this - particularly in one branch, with the observations being relatively brighter than the models at phases nearer Full Moon, compared to phases nearer new Moon.
Since we have seen the EFM method produce less phase-dependent albedos [see here: http://iloapp.thejll.com/blog/earthshine?Home&post=252 ] we think the Laplacian method needs further development and investigation before it can be used.
It is worth listing why the Laplacian method might be useful:
1) It does not require careful alignment of model and observation. The signal is extracted from a robustly defined location in each image.
2) It is a 'common-mode-rejecting' method and is not dependent on image resolution.
3) It is relatively fast.
Post-Obs scattered-light rem.Posted by Chris Flynn Dec 30, 2012 12:07AM
I have been looking at the use of GPUs versus CPUs for our scattered light analysis. We need to be able to convolve artificial Lunar images (outside the atmosphere) with the instrument PSF. GPUs offer a considerable speed advantage.First look (CPU versus GPU):
Upper left panel: artificial lunar image outside the atmosphere.
This artificial image is then convolved with a 2-D Gaussian-like PSF
which has fat (powerlaw) tails, and which closely reproduces what we see
in real data.
Upper right panel: convolution using 2-D FFT code running on a CPU
Lower left panel: convolution using 2-D FFT code running on a GPU
Lower right panel: the ratio of the two methods, i.e. the ratio of the two previous panels
There is a lot of structure in there, mainly images of the lunar
crescent turning up in different places -- at a level of about 0.1% of
IMPORTANT: the CPU code was written in double precision, whereas the GPU was in single precision.
(The above reproduces with more explanation an earlier post)
Notes: The CPU code calls the FFTW3 libraries from Fortran (Dec's ifort compiler is used), just using the standard Fortran to C wrappers provided with FFTW3. The GPU code is in written in CUDA. Second look (CPU only, single versus double precision):
The plot above shows the ratio of the single precision CPU versus
double precision CPU (i.e. no GPU results shown on this plot).
There is similar structure in the ratio -- and at about the
same level as the GPU tests gave, i.e. discrepancies at the level of a
few x 0.1% of the intensity.
Third look (CPU in double precision, renormalisation)
In this plot
we compare CPU double precision, applied to the ideal Lunar image, and without "min/max
renormalisation". (Min/max renormalisation means scaling the input image so that the smallest value in the frame is 0.0 and the largest value is 1.0).
The ratio panel of the two convolutions (bottom right) shows noise only, and at a
very low level -- 1 part in 1E7. Highly acceptable!Fourth look (CPU, single precision, renormalisation)
This plot shows the same as the previous one -- but with single precision rather than double. The artefacts are back, at the same old level of a few x 0.1%!Thoughts:
We might already be able to conclude from the above that double precision FFT/CPU is robust (negligible
artefacts), but that a single precision CPU, or a single precision GPU,
produces similar sized (few x 0.1%), and thus slightly worrying, artefacts.
But I need access to a double precision GPU to test this. Hope to do so next week!
The acid test will be the results of comparing double precision on a CPU to a GPU.
From flux to AlbedoPosted by Peter Thejll Dec 27, 2012 09:52AM
On this blog we have previously investigated the laplacian method of estimating earthshine intensity. [See here: http://iloapp.thejll.com/blog/earthshine?Home&post=273
and here http://iloapp.thejll.com/blog/earthshine?Home&post=272
One of the limitations we realized then was that image resolution influenced the results from the Laplacian edge signature - the more the edges of the image are smeared the lower the Laplacian signature estimate will be.
We now revisit the method after realizing that the whole image is affected in the same way by resolution issues - the intensity estimate derived from the DS and BS edges will both depend on image resolution in the same way. The DS estimate is proportional to earthshine intensity, while the BS estimate is proportional to 'moonshine' intensity (plus earthshine - a vanishing contribution). The ratio of the two estimates will be independent of resolution - i.e. the ratio has the 'common mode rejecting' property.
We convolved ideal images of the Moon with the alfa=1.8 PSF and estimated both the 'step size' at the edges (on DS as well as BS) and the Laplacian signature. We plot them below:The top panel shows the step size estimate against the Laplacian signature estimate on the DS only. We see an offset, with Laplacian estimate being smaller than the step size estimate. This is caused by the resolution issue - the edge is fuzzy and the Laplacian is degraded. The step size is estimated robustly from the original ideal image (i.e. before the convolution is performed) and is thus not dependent on image resolution.
The bottom panel shows the ratio of DS and BS estimates - x-axis is the step size estimator and the y-axis is the Laplacian estimator. We now see that the dependence on image resolution is gone from the Laplacian estimate. The correlation between the two estimates is good, at R=0.92, but not perfect. Noise was not added to the images. Estimates of Laplacian were performed on the line through the disc centre. The step size estimator is based on the average of 9 lines through the disc center.
[Added later:] Since we are working with synthetic models we know the actual earthshine intensity in each image, so we can compare what the actual intensity is to what is found with the 'step' and the 'Laplacian signature' methods:In the first panel we see what we saw above - that the step and Laplacian methods are rather consistent. In the panel below we compare to the actual earthshine intensity.
Since the Laplacian signature method (and the step size method) literally express the DS/BS intensities ratio (which is only proportional to the earthshine intensity) we have different values along the x- and y-axes. The geometric factors having to do with distances and Earth's radius are not compensated for.
The main result is that the relationship between actual earthshine intensity and the Laplacian signature method (and by extension, the step size method) are not quite proportional - there is a slight curve. In interpreting the above we should keep in mind that the Laplacian estimate of DS illumination from the edge derivatives is in a different role than the estimate of BS illumination - the former is not very geometry dependent while the latter is: That is, earthshine is due to light from a source we are sitting on doing our observations from, while the BS illumination is more angle-dependent in that the reflectance properties of thr Moon come into play to a larger degree - at the moment I am not sure whether there is an angle-dependence 'along the BS edge of the disc' that can cause a problem in interpretation. Will need to look at this.
Another thought: is it possible to make a calibration relation between actual earthshine intensity and the Laplacian DS/BS signature estimate?
Post-Obs scattered-light rem.Posted by Daddy-o Dec 19, 2012 02:07PM
We do have many problems to contend with - but now and then we are confirmed in what we set out to do: Here is an example of the scattered-light reduction of an observed image (2456034.114etc) where the EFM Method seems to be doing well compared to BBSO linear. We compare to a synthetic model image generated for the moment of observation (i.e. the libration and geometric factors are representative of the observing situation):Top panel: slice across the Moon at row 296 so that Grimaldi near the edge is transected. The black curve is the synthetic model unconvolved - it is in units of W/m². The red curve is the scaled and offset profile along the same row of the EFM-cleaned image; and the blue curve is the BBSO-linear cleaned image identically scaled and offset.
Middle panel: detail of the above.
Bottom panel: difference between EFM and ideal and BBSO-linear and ideal, along row 296, expressed in percent of the ideal value.
The observed data were scaled since the units are different; they were offset because the EFM model did not have 0 value on the DS sky. The BBSO-linear, being 'anchored in the DS sky' did have a 0 value in the sky. The same scaling factor and offset was used on EFM and BBSO, for comparison, however - hence the little offset on the DS sky at right.
What do we see? Well, the EFM-cleaned (red) line follows the synthetic model quite nicely between column numbers 325 and 370 while the blue line (BBSO-cleaned) diverges all the time. Near the DS edge (columns 370-385) the synthetic model is higher than EFM.
What does it mean? The BBSO-linear method has better removed the flux on the sky - it is designed to do that, while the EFM is designed to minimize the residuals squared in a mask on the sky around the Moon. This implies that the BBSO-linear method, in the present case, would be better than EFM on the lunar disc near the DS sky. As we move further onto the DS disc the BBSO-linear method will fail more and more since the halo is not linear with distance from the edge - the method underestimates the amount of scattered light on the disc between the edge and the BS. We do see how the blue line diverges more and more; we do see the red line cling closer to the 'true value' (assuming the synthetic model is 'true') across the disc, before it too fails nearer the halo and the BS. The behavior nearer the edge may be a consequence of how we model the synthetic images - we have to use a reflectance model to make the synthetic images - and the angles of emergence and incidence corresponding to 'near the edge' is not one for which the reflectance model is inherently very good. The models we use are based on the simple 'Hapke 63' model. We speculate that the disparity between observations and synthetic models in the columns 370-385 is due to model inadequacy.
Anyone using a reflectance model as simple as the Hapke 63 model will encounter the above problem if they try to use pixels near the lunar disc edge - the natural thing to do, if the halo is removed using the linear method is to use edge or near-edge pixels.
Hence the problem of reflectance modeling and the inadequate linear method become coupled! Our EFM method seems to be a way around this obstacle - allowing use of disc areas further from the edge where simple reflectance modelling is adequate - hence we should be getting more reliable terrestrial albedo data. One day.
Post-Obs scattered-light rem.Posted by Daddy-o Dec 19, 2012 08:50AM
In the EFM-method we determine the alfa values for every image. Is there a link between the alfa value for one filter and the rest in an interval of time? It is our understanding that alfa is determined by the amount of scattering in the optics plus the atmosphere. We therefore expect that on 'bad' nights the alfa values will tend to move in the same direction. We investigate this here.
We find all alfa values in all EFM-treated images. We sort them into half-hour bins. We calculate all the alfa values in each bin and plot the results. Below is a pdf showing all the plots between some filters, at different 'zoom-levels'. The image shows the last zoom-level, highlighting the dense 'clump' of points:
There seems to be a general agreement that the alfa values are correlated - bad nights (i.e. 'broad PSFs') occur in all filters at the same time. Since VE1 is just about identical to IRCUT the scatter seen above means that the fitting routine is unable to make a perfect match - or that observing conditions, during the half-hour bins used, changed.
Using 15 minute bins does not improve matters:
I therefore suspect that the fitting method does not find the best fit each time.
From flux to AlbedoPosted by Daddy-o Dec 17, 2012 02:19PM
What B-V colour should we expect for the earthshine?
We will here estimate it by using the change in colour of Sunlight that has struck the Moon once, and the colour of Earth as estimated from spacecraft.
The Sun's B-V is +0.650 [Allen, 1973] [Holmberg et al, MNRAS, 367, 449, 2006
The Full Moon's B-V is +0.85
[Lane&Irvine, AJ 1976 78, p. 267]
[vdBergh has +0.876
for Mare Serenitatis;
Allen 4.ed. table 12.16 has 'Moon' B-V 0.92
Gallouet (1963) has +0.94
Wildey & Pohn (1964), AJ vol 69, p.619 have a range of values near +0.86 to +0.87
(their work seems good and a milestone).]
The Earth's B-V is 0.2 [Allen 3. ed, but appears based on a 1961 work - so pre-spaceage?]
The Moon's DS B-V is 0.64 on average given data in [Franklin (1967), JGR 72, p 2963]
If Sunlight is reddened by one reflection off the Moon by 0.85-0.65=+.2 mags, then we expect earthshine, bounced once off the Moon to redden by the same amount.
If the Earth has B-V=0.2 as seen from space then seen after one reflection it ought to be redder by +0.2 or appear to us observing it from Earth at B-V=0.4. This is not what Franklin measured.
Basically, we do not yet know Earth's B-V colour! I am making inquiries, and we shall see.
Note that Danjon did lots of colour observations of earthshine - but in the Rougier system. Wildey [JGR vol 69, p.4661+] refers to a transformation from Colour Index ("C.I.") in the Rougier system to B-V in the Johnson system - but without giving numerical details. The method is based on transformations using the Full Moon and the Sun colours.
The transformation should be made specific and the data from Danjon placed online. Another student project!
Post-Obs scattered-light rem.Posted by Daddy-o Dec 17, 2012 11:34AM
We compare the B-V values on the DS of images from JD2456034: we look at images only exposed to bias-removal ('DCR' images) and images cleaned with EFM and images cleaned with the two variants of the BBSO method: linear and log. We show a 'slice' across the disc at row 256:Top left: black is the B-V slice from the DCR images; red is the B-V values from the BBSO-linear cleaned images. The second graph in the upper left panel is the difference between the B-V values (BBSO-linear images minus cleaned image). The vertical dashed lines show the edge of the lunar disc and the start of the BS in column units.
Top right: same, but for BBSO-log method.
Bottom left: same but for EFM images, as shown elsewhere on this blog.
The results are quite different - the BBSO-linear method has given us fairly constant B-V values across the disc - they are about .05 mag below the DCR values (i.e BBSO-linear are bluer than DCR values).
The BBSO-log values seem completely unrealistic.
The EFM-cleaned values also look a bit unrealistic in that there is a spatial dependence on the magnitude of the B-V relative to DCR - in a way that looks like a remnant effect of the halos. The Delta(B-V) value changes sign across the disc.
The night JD2456034 is very close to New Moon and we know from other results that this is when the BBSO-linear method is likely to work best (the halo being small). Since there is a phase-dependency in the overall results for BBSO methods over and above what EFM shows we know we cannot universally use BBSO results. On the DS, towards the edge, the BBSO-linear method should be very good - it is 'anchored in the sky', unlike the present EFM method, and therefore should be unbiased near the edge. Our EFM method, at the moment, only minimizes the square of the residuals on the sky, inside a mask.
Near the sky, the DS B-V values in the BBSO-linear and EFM-cleaned images differ by about 0.04 mags. We should look at EFM methods that also 'anchor in the sky' and see what we get then.
The present largest worry is not the B-V offset but rather the dependence on position on the disc that the EFM method shows.
What does the literature tell us we should expect B-V to be under earthshine?
At the moment I only know of Franklin's 1967 paper. It gives B and V values of the earthshine.http://adsabs.harvard.edu/abs/1967JGR....72.2963F
The average B-V seems to be +0.64. Can we find any other information on B and V?
Showcase images and animationsPosted by Chris Flynn Dec 14, 2012 06:22AM
I have made four colour maps now in B-V on four different nights.
The nights are arranged like this:
A uniform scale is used for all four images, from B-V = 0.0 to 1.4. The BS light is coming out in all four images around B-V=1.0, and the ES at around B-V = 0.6-0.8,
depending on whether one is looking at highlands or lowlands (lowlands are redder).
The cause of the black dips along the BS rim is the problem that the B images are slightly larger than the V images - so even with good registration of the center of the moon (to the closet pixel), there are issues in producing these colour maps. The falloff in the halo light cannot be the same power law in V and B, because the halo changes colour -- but this is still to be checked.
Observing logPosted by Chris Flynn Dec 14, 2012 05:58AM
I wrote a program called moonraker in fortran, which processes all our data, seeking out the best for further reduction. Peter has written something similar in IDL.
It goes through all the images and throws out ones with
1) saturated BS (> 50,000 counts in the peak of the BS)
2) underexposed BS (<10,000 counts)
3) smeared images (heavy bleeding of photons in the Y-direction on the chip, reaching the bottom 20 pixel wide strips of the frame). A mean count level of more than 50 counts in this strip triggers a flag and the frame is rejected.
4) rough check if the moon is too close to the edges of the frame. This is not yet optimal, but throws out the worst cases.
5) checks that the temperature of the CCD is OK (an entry in the fits file header tells us if the temperature of the CCD has stabilised or not)
If all 100 slices in multiple exposure frames are OK, according to the above, they are bias subtracted (using the biases on either side of the exposure, averaged appropriately for each slice, and scaled to the superbias created by Henriette) and averaged, and written to disk.
6) The magnitude of the total light in the frame (i.e. apparent magnitude) is then compared to the expected V magnitude (using the JPL model, with corrections for the east and western sides of the moon, depending on which side is illuminated by BS), taking into account the airmass of the observation and the extinction for the particular filter. This isolates for removal exposures which have been taken with an incorrectly reported filter. Finally, after extinction correction, only those frames which fall within ~ 0.2 magnitudes of the correct magnitude using the JPL model are retained.
The list of the final set of good exposures using this method is attached (all100.doc -- actually an ascii text file):
There are 534 good frames, spread from nights JD2455938 to JD2456104.
A summary is as follows:
Night, number of good frames in V, B, VE1, VE2, IRCUT, comments
JD2455938 2 2 0 5 0 fullish moon, some foggy frames
JD2455940 3 1 0 2 0 fullish moon
JD2455943 3 3 0 3 1 fullish moon
JD2455944 3 1 2 1 3
JD2455945 1 1 1 2 0
JD2456000 3 0 0 1 1
JD2456002 7 0 0 0 0 V band only
JD2456003 9 0 0 0 0 V band only
JD2456004 1 0 0 1 0
JD2456005 6 0 0 0 0 V band only
JD2456006 4 0 0 0 0 V band only
JD2456014 3 0 0 0 0 V band only
JD2456015 7 10 4 6 4 1/3 moon, lots of frames
JD2456016 11 10 9 11 11 "
JD2456017 7 11 0 11 2 "
JD2456028 5 5 0 9 0
JD2456029 0 0 0 6 2 2/3 moon, lots of scattered light
JD2456030 1 0 0 3 2
JD2456032 1 0 0 4 1
JD2456033 4 1 4 2 4
JD2456035 0 0 0 1 0
JD2456045 5 5 4 4 4
JD2456046 4 8 10 11 10
JD2456047 6 5 8 6 7 stars visible in some frames, useful for PSF
JD2456061 2 4 3 6 4
JD2456062 0 0 0 2 0
JD2456063 0 0 0 5 0
JD2456064 0 0 0 3 0
JD2456073 4 7 7 10 8 some eclipsing of moon - dome issues?
JD2456074 4 2 3 7 5 scattered light haze or fog?
JD2456075 8 7 8 7 8
JD2456076 9 1 0 2 0
JD2456089 3 2 1 1 0
JD2456091 2 1 5 0 4
JD2456092 0 0 3 0 5
JD2456093 0 0 1 0 5
JD2456104 5 7 5 1 5 some foggy frames, moon a bit close to bottom corner
I've inspected all of these by eye and they look mostly pretty good! There are still a few bad frames here because the scattering of the halo is not selected for yet -- but the number of frames with low alpha (i.e. haze or fog) is just a handful.
Some of these nights will be pretty useful for looking at colour changes in the ES over time. I am doing that next.
Showcase images and animationsPosted by Peter Thejll Dec 13, 2012 10:06AM
In this - uipdated - posting we combine more images from JD2456034 - we construct B-V images in various ways.
The discussion refers to Chris' originalposting:http://iloapp.thejll.com/blog/earthshine?Home&post=275
The first images we look at are not the bias-reduced images Chris uses, but 'EFM-cleaned' images (i.e. scattered light has been removed). Both the B and V image used were co-additions of 100 images each. The two images were centered in the image frame. We look at tow sets of B and V images - the one at the top is cleaned with one setting of the EFM method; and the one below is done with another setting of the EFM method. Some of the features to the left on the sky are the effect of 'cyclical overlap' from the right side when the image is shifted.
We see the DS to the left and the BS to the right. The BS halo and the BS have become undefined (i.e. 'NaN') because either B or V is negative here (remember, it is not a bias-reduced image with full halo in place - these are EFM-reduced images so the BS and parts of the halo are now zero or negative. In overlap almost all of the BS and the BS halo have become NaNs!
Here is a 'slice' across the middle of the upper image:
And here is the same slice across the lower image:
And finally, here is the slice across the un-reduced B-V image - that is, the image formed from B and V images calculated from the images that were only boas-reduced (as in Chris' plot).
The uppermost image has a large 'dip' when we get near the BS and its halo remnant . In the middle image the halo has apparently been better removed. STrangely, thereis least sign of a halo in the 'raw' images where nothing has been done to remove the ahlo. This needs to be discussed!
We used other airmasses than Chris. In my calculations the airmasses for the two images involved are:
B_am=2.545 image is 2456034.1142920MOON_B_AIR_DCR.fits
V_am=2.477 image is 2456034.1164417MOON_V_AIR_DCR.fits
I used kB=0.15 and kV=0.10, like Chris.
I did not solve for B and V by solving two equations with two unknowns - I iterated. Convergence was fast. I iterated on the whole images.
The first of the above EFM cleanups was done with a weighting of the mask used to define the area of the sky on which to reduce the sum of squares that favored the DS part of the sky. In the second attempt above equal weight was given to the RH and LH sides.
Showcase images and animationsPosted by Chris Flynn Dec 13, 2012 04:42AM
Following on from the previous post, we have made a colour map based on two images (V and B) taken on JD2456034
Both are averaged results from stacks of 100 exposures, so the S/N ratio is 10 times better compared to the images in the last post.
V band exposure time was 72.5 ms, and the B band exposure was 222 ms, for a total of 7.3 seconds in V and 22 seconds in B. The observations were at an airmass of ~ 2.3.
The colour map looks like this:
with the colour scale (B-V) running from 0.2 to 1.3.
The BS (brightside) has a colour of around 1.0, which is still a little redder than the expected value of 0.9 (for BS on a full moon) by van den Bergh (1962)
There is more scattering of the light in B than in V, which accounts for the deep valley to the right of the BS, and the excess halo light beyond it. Colours here are not reliable. Colours just beyond the concave edge of the crescent on the DS are probably also affected by differences in the scattered light profile, so should be regarded carefully (this part of the moon can be examined when the crescent is on the other side). The left half should be pretty good!
The DS (darkside) has colours ranging from about 0.6 to 0.9 -- with the lowlands redder than the highlands.
Data reduction issuesPosted by Chris Flynn Dec 12, 2012 06:00AM
We have used B and V images of the moon from JD2455859 to make a map of the B-V colour of the reflected light from a thin crescent moon.
We use the photometric transformations derived from our open cluster observations: http://earthshine.thejll.com/#post229
The B band image of the moon was shifted to the position of the V band image, and the colour computed from the fluxes in B and V directly from the pixel fluxes.
The map below shows the result:
The scale at bottom runs from B-V=0.1 to B-V=1.5.
The darkside (DS) has a B-V colour in the range 0.6 to 0.9 -- whereas the crescent has B-V ~ 1.1. The measured colour of the BS is B-V ~ 0.9 (Van den Bergh, 1962, ApJ, 67, 147
), so we may be a little too red. More work needed on this.
We are seeing roughly the colour difference expected between ES and the BS -- ES should be a few tenths bluer in B-V.
There is a clear residual halo in the colour map, indicating that the scattered light around the crescent falls off differently in B and V filters (as we expect).
Very interestingly, the crater pattern becomes much harder to see in the colour map. The V and B images look like this (after registration) V on the left, B on the right:
The lowlands appear to be slightly redder than the uplands in the colour map, by a few tenths of a magnitude in B-V.
More work needed on this -- this is just a progress report!Technical info:a)
Images used were:
2455859.1313477MOON_V_AIR_DCR.fits and 2455859.1365256MOON_B_AIR_DCR.fitsb)
The transformations used were (airmass is 2.6 for V and 2.7 for B, extinction coefficients for V and B are 0.10 and 0.15 respectively).
Vinst = -2.5*log10(sumv/exptimev) - 2.6*0.10 ! airmass = 2.6
Binst = -2.5*log10(sumb/exptimeb) - 2.7*0.15
V = Vinst + 15.07 - 0.05*bmv
B = Binst + 14.75 + 0.21*bmv
which implies that B-V = 1.35 * (Vinst-Binst) - 0.43
(the airmasses above were incorrect in the first version of this entry, they have been corrected now)
Post-Obs scattered-light rem.Posted by Daddy-o Dec 10, 2012 04:31PM
In this entry on the blog: http://iloapp.thejll.com/blog/earthshine?Home&post=272
we discussed the Laplacian method of estimating earthshine intensity by removing the influence of the scattered light halo. We have tested it on a good image from near New Moon (i.e. lots of bright DS, very little interfering BS). Consider this plot:
In the top two rows we see the observed image, the EFM-image and the Laplacians of these. In the second row second plot there are three additional curves, in black. The centre one shows the estimate of where the disc edge is, based on the estimates we have of disc centre and disc radius. The other two lines are parallel outliers at + and - 40 pixels from the edge. In the third row we show first the coaddition of all rows along the edge, using the estimates of the edge and the outliers as start and stop - this allows a coaddition that centres on the disc edge. Second in third row is the same thing based on the Laplacian of the observed image. We see the signature of the double derivative of an edge. In the last row we show the same as in the third row but for the EFM-cleaned image and its Laplacian.
The application of the EFM method removes most of the halo and this is consistent with the slight lowering of the slope after the 'step' in the first panel of fourth row, compared to first panel of third row. We extract the size of the 'jump' as well as the distance between the minimum and the maximum of the Laplacian edge signatures. The jump size is estimated as the difference between the mean of the first third of the curve and the last third. We collect the four estimates of the earthshine intensity - two jump heights and two estimates based on the Laplacian signature, and plot them below for 4 differently rebinned versions of the original images - unbinned; binned 2x2; binned 4x4 and binned 8x8. This is done to 'sharpen' the edge in case the edge is 'fuzzy' so that the 3-point numerical operator used in constructing the Laplacian will interact correctly with the edge.
We see color-coded graphs. The estimates of earthshine intensity based on the observed image and the EFM-cleaned image are black and blue, respectively. The estimates based on the 'signature size' in the Laplacian of the observed and EFM-cleaned images are red and orange, respectively.
The differences are quite large - several percent, and there is a dependence on the rebinning of the image - even in the case of the step method. The results that 'ought' to be the best is the jump-size one based on the EFM-cleaned one since the halo is absent and because the jump size method should have less statistical error-of-the-mean than the signature size method since more pixels are involved in estimating the former. The error-in-the-mean for the jump estimates for the unbinned image is of the order of tenths of percent - clearly dominated by systematic effects.
The above suggests that there are large method dependencies.
The 'jump estimator' may be victim of the slopes seen near the jump - especially in the EFM case - with such slopes the mean before and after the jump is dependent on just how you estimate the jump. Possible improvements is to extend a linear regression from the first half or third of the points, before the jump to the midpoint and do the same from the other side and calculate the 'jump size' as the difference between the two extrapolations. The slope is still a concern, however. It is due to the lunar surface albedo variations. The Jump size itself is a function of earthshine intensity as well as lunar surface albedo and therefore on libration, as well as the geometric distance factors. Being estimated right at the edge the reflectance is a constant, leaving lunar albedo as a variable factor (through libration). This could be taken out with knowledge of the parts of the surface of the Moon right at the edge - i.e. via modelling.
The 'signature size' method has different challenges - it is based on fewer pixels for one thing. It also depends on the lunar albedo - and not on reflectance, provided the edge is used (and not features on the DS, as Langford et al do). However, the method depends on knowing that the image is in good focus. Bad focus will blur the edge causing a lowering of the signature size - for area methods such as the EFM we do not depend on a sharp edge but can estimate levels from regions chosen away from the edge. Of course, we still get blurring from lunar features overlapping.
The above study did reveal dependence on binning - but if blurring was being addressed the results are not encouraged as there was no convergence.
So many things to compare! Super student project! A study of artificially blurred synthetic images could be done, and 'step' and 'signature' methods compared.
Post-Obs scattered-light rem.Posted by Daddy-o Dec 10, 2012 11:29AM
In an interesting article Sally Langford, et al. [ http://adsabs.harvard.edu/abs/2009AsBio...9..305L
] describe an analysis method for earthshine images. Essentially, the Laplacian's effect on images is to detect edges and the size of the 'jump' in going from e.g. sky to lunar disc in the observed image translates into an amplitude of a characteristic signature of the derivatives of an edge. This enables the large-spatial-scale features of the scattered light halo to be removed from the short-spatial-scale features of edges and craters.
Our method for removing effects of halo is to construct a forward model of the halo, based on the BS of the Moon (caught in the same image as the DS - Sally's FOV is smaller - about 20 arc minutes; our is 60 arc minutes), and subtracting it, leaving the DS almost halo-free.
How well does the Laplacian method work on our type of images? Sally's images were well-exposed images of the DS alone - allowing large counts - 10's of thousands; ours are co-addition of 100 images taken so that the BS is not overexposed, leaving the signal on the DS typically near 10 counts. One image with 10,000 counts should thus have the SNR of 100 of ours.
We took one of our good images - from JD 2456095 - near New Moon - and plot the profile across the image near disc centre in the image that we obtain using the forward modelling method (named 'EFM' in the plot and this blog) and the same profile from the Laplacian of the image. We average 20 rows.
Inspection of the columns (approx # 100-120) where the lunar disc edge is we see a clear 'step' up from the sky level in the EFM-cleaned image but there is no sign, above the noise, in the Laplacian of the same image. Inspecting the whole Laplacian of the EFM image we do see a faint signature:
The above image is 'histogram equalized' to show the feature. Our own method - also shown when histogram-equalized is here:
We see that the halo has been only partially removed on the right.
I think we would be hard pressed to extract a signal from the Laplacian of the image. It also bothers me that the whole earthshine signal is reduced to the value of the derivatives in just edge pixels. In our own image we have hundreds of pixels to measure on - in the Laplacian we get only a signal along the edge.
We should note that the remnants of halo on the right are much less evident in the Laplacian image, suggesting that there is a kernel of a good idea in the method. For reference, the Laplacian of the raw observed image is here (histogram equalized):
It does seem as if the Laplacian helps remove a lot of the halo, but also reduces the analyzable part of the image to the DS edge.
The Langford paper mentions both smoothing images and co-adding them - the former is done at the resolution of the worst image for a sequence. The Langford paper analyses features on the disk - not the edge.
Working out how the Laplacian is best used, on realistic images of the type we have, would be a good student project!
Post-Obs scattered-light rem.Posted by Daddy-o Dec 04, 2012 04:23PM
We have, below, studied the 'structures' or 'bands' that, in some images, stretch from side to side. One concern was that the structure was induced by some step in image reductions. We therefore compare raw images to EFM-cleaned images to see if the structure is present in both.
Visual inspection of histogram-equalized versions of raw and EFM versions of the images reveals the structure, although it is easier to see in the EFM image since the halo and most of the BS is gone:
Plotting the average of the first 20 columns of each image, and scaling them to each other shows that the structure better:
The black line is from the EFM-cleaned image, while the red curve is from the raw image.
This seems to rule out that the 'structures' are induced by image-reduction steps.
We remain suspicious of the possibility that the bright light from the BS itself somehow is causing the problem. It is not a reflection since the structure is a deficiency of counts. Only way to explain it is by some process subtracting counts from these bands - or lowering the sensitivity of the whole set of bands - perhaps some sort of non-linearity caused by strong light in one part of the CCD affecting the whole row?
We still need to map when the structures is present - we want to see if it appears in a timeline or is a function of the placement of the image in the frame. Inspecting the first 700 images of 2000 for the EFM frames shows no sudden dependency on time of this problem.
Idea: also extract the maximum counts of the image and use this as a parameter - we do have suspicions that non-linearity sets in far below 55.000 counts.
Question: Why is the structure along rows, while readout direction is along columns? What is special, in a CCD chip, about the rows?
More to follow.
Bias and Flat fieldsPosted by Daddy-o Nov 30, 2012 02:43PM
In the entry below, at:http://iloapp.thejll.com/blog/earthshine?Home&post=269
we considered the need to remove some residuals left over when a poorly-fitted halo had been removed from the observed image. Sensing that the problem has to do with asymmetry in the solution forced by the lunar disc not being well centred, we consider now the effect of lunar position in the image on the quality of the halo fit. We estimate the quality of the halo fit from the mean value in a sky-patch near the DS.
We see here on the x-axis the value of the disc centre coordinate (i.e. column number in the image) and on the y-axis the mean value of the DS sky patch. We seem to have some scatter as well as a structure that looks like an inverted parabola, for these points near y-value 0. That is - the sky-patch mean value of the DS in images where the halo has been removed with the present EFM method depends on where the lunar disc is - the further away from x0=256 (middle of image) the Moon is, the larger a residual is left on the sky after the halo is removed with the EFM method.
We need to invent a better EFM method!
Bias and Flat fieldsPosted by Daddy-o Nov 30, 2012 08:45AM
In this post:http://iloapp.thejll.com/blog/earthshine?Home&post=268
I pointed at the unwanted structure in the sky near the lunar disc in an image that was Bias-reduced as well as had had its halo removed. Bias problems were ruled out - and left was a speculation on internal reflections in the camera system.
If we are truly left with that horrible structure on the sky we cannot just remove the halo in the present way, as the method depends on 'fitting the sky'. If there are basins and hills in the 'sky' the fit will be bad - if the halo subtracted has the right shape but not the right 'level' then perhaps we can think of a fix: After removing the halo, but before extracting photometry from the DS disc in the image we can reference the sky adjacent to the DS and offset our disc value from there.
I tried this, by estimating the sky level on a part of the sky near the DS on an image where the halo has been subtracted, following the EFM method:
The two patches on the lunar disc are the Grimaldi and Crisium reference patches (DS and BS, respectively [although the BS value is estimated from the same pixels in the raw image]) and the large semi-rectangle on the sky next to the DS is the 'skypatch reference area'. We calculate the average value of that patch and subtract it from the value extracted for the DS. If the error is of an offset type, rather than a slope type we have then corrected for the halo misfit.
To see the effect of this problem we estimated the DS/BS ratio in all EFM-cleaned images, for each filter, with and without referencing to the sky level:
As usual, the jpegs above are poor in quality so we also post the pdfs:
Each panel contains three columns - the rows are for filters. The first column is the DS/BS ratio as function of lunar phase (Full Moon is at phase 0). The second column is the ratio of the DS and the total RAW image flux [for reasons explained elsewhere!]. The third column shows the 'alfa' value derived by the EFM method.
Almost the same list of images were used for the two plots above - note the differences by inspecting the alfa plots - crosses are present in one plot but not the other one.
We see that the sequences of points for the case 'skyptach reference level removed' are slightly fuzzier than the sequence where the reference is not subtracted! Before we can interpret this I think we need to restrict the two plots to the same set of images - I shall do this and return ...
[later]: This has now been done. For the IRCUT filter (341 data points) we show the effect of skypatch-referencing on the DS/BS ratio expressed as obs/model:
First we notice the wide span in DS/BS along both axes - this is not new: this is mainly the phase-dependence we see - this is MUCH LESS than would be seen for BBSO-method treated images, and is not the point - the point is that the spread along y for a given value of x - say near x=1 is something like 50%: That is - the effect of removing the sky-level from the DS value, in EFM-cleaned images, is to alter the DS/BS ratio by 50%. This means that the EFM-method did not do a very good job of removing the halo. It did remove a lot of the halo - but clearly not so much that a considerable effect is felt when the small remaining offset is removed.
This is important for our Science Goals: On the one hand we see the EFM method work much better than 'BBS linear' and 'BBSO log' methods in removing the effect of phase on the DS/BS ratio [shown elsewhere in this blog], on the other hand we see it does not do a sufficient job.
We should note that the BBSO linear method removes the sky level in one step (since it is a fit to the sky near the DS) but that, being linear, it does a poor job of removing the halo where it matters most - on the DS disc. The EFM method may be much better at removing a halo of approximately the right shape - but there remains a bias that is important.
The current EFM method is not 'anchored in the sky' - it merely seeks to minimize the residuals formed when an empirically generated halo is subtracted from the observed image - as evaluated on a masked section of the sky part of the image. There are choices made when that mask is set up, and they are:
a) The mask used above was such that sky was included on both sides of the lunar disc, set off from the disc by some 20 pixels and curtailed vertically at 0.7 radii (so that a 'band across the lunar disc, but not including the disc' is set up). This gave weight to both the part of the halo on the BS and the DS.
b) The mask is quite large to give access to a large number of pixels so that the effect of noise on the fitting-procedure was reduced.
The requirement in b) forces the choice in a) to be so large that any unevennesses in the sky (such as we suspect occurs due to internal reflections when the Moon is near the side of the image) has an influence on the quality of the fit, forcing the above consideration of a 'special removal of the sky level near the DS'. To this comes the problems of allowing the BS halo to influence the fit.
We should investigate how we can improve the EFM method. Possibilities include
1) spatially weighting pixels so that 'difficult areas' are avoided or are given less influence on the final fit - such as the bothersome BS part of the halo.
2) use much smaller areas to fit the halo on the DS to - such as a patch near the DS patch on the lunar disc. This has to be balanced - on the one hand we get more influence of noise (which influences the BBSO method too, as it uses 'narrow radial conical segments' on the sky), but on the other hand we eliminate the effect of the BS halo as well as the 'unevennesses in the sky field'.
3) One hybrid form of the EFM, tested earlier, consisted of enforcing that the model halo should be flux conserving (as now) and at the same time pass through an average point on the sky near the DS in the observed image.
We should implement a method that does not show the above sensitivity to removal of a sky reference level calculated on the already cleaned-up image. Access to a system without these internal reflections would also be nice. One significant step to take before that could be to find all images without the 'streak' effect that lead to the above considerations.
Bias and Flat fieldsPosted by Daddy-o Nov 27, 2012 03:25PM
A strange structure has been found in some reduced images. To the right we show a dark frame taken seconds before the image on the left. The image on the left is an EFM image - i.e. the halo from the BS has been subtracted after fitting to the sky areas of the image. We clearly see the structure in the left image - a 'band' stretching to the left of the DS. This structure is not present in the dark image - so it is not an artefact of bias subtraction [Of course - the bias subtraction is not performed with adjacent dark frames - noise would be added that way - rather, a scaled almost noise-free superbias is subtracted. It is like a lower-noise replica of the image on the right - i.e. no structure, just mild level of noise.]. Below the two we show a plot along a vertical axis in the two images: columns 50 to 150 were averaged and shown in the plot as the black and the red lines.
There is a very clear structure in the sky of the Moon image. Given that it is not induced by bias-subtraction it must be due to the presence of the Moon itself. We speculate that:
a) it is some optical effect - reflection - from the inside of the camera or telescope. Halo-subtraction has reduced the sky level to almost zero but a little too much has been subtracted in the 'dark band' and a little too little has been removed outside the band. The fitting of a rather smooth 'halo' from the observed image could give this effect, if the structure itself is present in the image.
b) some electronic effect is causing the rows with the very bright BS in to somehow 'jump low' due to some effect we do not understand. NB: The readout direction is 'down' - not to the left!
Note the presence of what looks like a truncated halo to the left in the image, at the frame edge: this could be an internal reflection showing the right hand side of the halo being reflected inside the camera. We saw this same effect in the animated sequence of the tau Tauri occultation - as the Moon migrated to the RHS of the image frame an 'echo' appeared on the left.
This therefore seems to support idea a) above. If this is the case we may have an effect on solution-quality from position of Moon in the image. We should investigate if there is a position nearer the centre that eliminates the effect, and then omit images that are too close to the edge.
Data reduction issuesPosted by Daddy-o Nov 23, 2012 09:06AM
One of the reduction steps performed has to do with the scaling of the bias images due to the thermostat-induced temperature variations in the CCD chip. This temperature variation causes a 20 minute period in the mean level of the bias with an amplitude of almost 1 count - thus of importance to our attempts to analyse extremely small signal levels.
We take Bias frames on both sides of all science exposures - one just before and one just after. If we were to just subtract the average of these frames from our science image we would be adding noise to the result - we therefore need to subtract a smoother bias frame. We have constructed a 'super bias' frame as the average of hundreds of bias frames - it is very smooth, but probably has a level that is unrelated to the actual level in each frame.
By scaling the super bias to the mean level of the average bias frames taken at the time of the science frame we get a scaled superbias that we can subtract - it has the right level and very little noise.
We need to understand how the scaling procedure performs, so we have extracted the scaling factor from the 5000+ exposures we have.
Top frame shows the factor on the superbias as a function of the sequence number, and the bottom panel as a a function of the Julian Day of the observation.
Most factors are near 1 but some stand out. 9 files have factors above 1.09 - their Julian days (integer part) are:
2455938 (7 images) 2455940 2456032 (one on each).
A list of the 209 images with factors over 1.08 is here:
The 4 unique JDs are: 2455814 2455938 2455940 2456032, with the majority of cases on 2455814.
These images should perhaps be inspected very carefully for problems with bias.
A close-up of the factors nearest 1 looks like this:
Data reduction issuesPosted by Daddy-o Nov 22, 2012 10:28AM
Using the methods described by Chris in the entry below - i.e. at http://iloapp.thejll.com/blog/earthshine?Home&post=265
I have also selected for the likely good data by taking extinction into account and looking for linear sequences. This resulted in a set of images that I deem 'good'. That list can be compared to Chris' list.
Chris has 3162 'good' images, while I have 2990. The cross between the lists finds 2273 instances on both lists. This list of 'jointly agreed good images' is here:
Methods for selecting good images differ slightly: Chris does not yet consider the alpha value found in the EFM method - I select for alphas in a narrow range near the mode of the distribution for each filter.
With converging selection criteria the list above would expand somewhat - perhaps best to keep the criteria a little different to avoid duplication of potential errors.
The joint list contains 49 unique nights. The distribution of images over nights is:
where the first column gives the number for the night. Note that towards the end of the sequence there are few images per night - that is because we were realizing the necessity to observe 'stacks' rather than sequences of single images which results in fewer co-added images.
A histogram of these data is shown above.
Data reduction issuesPosted by Chris Flynn Nov 21, 2012 11:35PM
We have been using the apparent magnitude of the moon compared to the JPL model of its expected magnitude (V band), to check which images suffer from gross problems, such as an incorrect filter in the beam, or light losses from thin cloud, or other reasons. This follows up on previous posts (http://earthshine.thejll.com/#post256
This plot shows the difference between the measured apparent V magnitude of the moon and the JPL magnitude (i.e the difference between instrumental and true magnitude) as a function of airmass. There are about 5000 frames, mainly of averaged data (i.e. up to 100 images averaged together) in the total data set (as of November 2012), of which about 1000 are in the V band, and plotted here.
(a pdf version is also available).
Blue represents one side of the moon (the side with the Crisium Mare), green the other side. Opposite sides of the moon have differing distributions of uplands and mare, leading to about a 0.15 magnitude offset between the reflected light and the JPL model (which implicitly assumes a uniform disc). The dotted lines show the expected trend with airmass for an adopted 0.1 magnitudes/airmass extinction -- quite close to what we are seeing in the data. There is a lot of structure in this diagram!
To clarify which side of the moon is which: the next plot shows the Crisium side illuminated by sunlight:
This plot shows the Grimaldi side being illuminated.
Now we concentrate on the crescent light on the Crisium Mare side (blue points in the plot):
Concentrating on just this side of the moon, we see a very tight (extinction) relationship with airmass, with notable outliers. They are all the the dim side of the relationship -- indicating loss of light relative to the JPF magnitude). The reason for most of these has been identified -- most often because of thin cirrus or haze (this affects the scattered light around the bright side of the moon, which we model with a power law with slope "alpha" -- these are indicated on the plot with that symbol. The moon went behind the tower and cables on the western horizon in one case "cables" and one image has low S/N. Two have "?" -- no reason was found for these outliers, so they are simply dropped. This leaves us with a nice looking sequence of data for this side of the moon. I'll look into these further as there may still be structure in there as a function of lunar phase.
The other side of the moon, the side with Grimaldi and extensive Mare, shows much less clear trends with airmass than the other side.
The dotted lines are the same as in the previous figure -- they bracket the good data on the Crisium side. Strange things are happening at low airmass (z<1.2) - it turned out that most of these peculiar data are from a single night (JD2455912
) as the next figure shows (background of these points shown in blue). Note the airmass scale has changed to make these points easier to see.The entire night of JD2455912 seems to be a dud
-- as similar strange happenings show up in the other filters -- VE1, VE2, IRCUT (but not clearly in B, as only one good B image was taken on that particular night).
Analysis of the remaining (grossly) discrepant points shows that they are often all on a single night. The next plot shows some of these problem nights:
Blue: Totally discrepant data from JD246691 : drop these.
Red: Near full moon!
Purple: data from JD2456061,2,3 (three nights). For almost all the moon is too bright by 0.3 mag -- possibly bias issues? Will look at this.
The bulk of the green points lie along the expected extinction line, with an offset relative to the other side of the moon (Crisium) of about 0.15 mag. We can bracket these data and use them for the ongoing analysis of earthshine. We look at this in the remainder of this post.
The next plot shows both sides of the moon in green and blue crosses, but for all filters (instrumental magnitude - JPL V-magnitude) as a function of airmass. No data have been dropped yet -- it's all the nights we have.
The colour of the moon in the various bands shows up nicely as an offset relative to JPL V-magnitude (e.g. B-V is of order 0.9, second panel from bottom). The Crisium side of the moon shows much cleaner results than the Grimaldi side, in all colours. The dotted lines bracket the good Grimaldi data, and are the same in all panels. Data which appears between these two lines in other bands then V, mean that the V filter was actually in the beam, not the nominated filter. There may be similar effects for all filter combinations, but for the moment we can oinly pick out misfirings which resulted in the V-filter being in the beam. Only a small fraction of the data seem to be so affected, which is very good.
Extinction depends on the wavelength of the filter : this is clearly seen as differing slopes in the zeropoint (ie. the difference between instrumental and true (JPL) magnitude) with airmass.
Once the grossly erroneous observations/nights are cleared out of the data, the sequence of good data as a function of airmass can be picked off the plot (we call this tunneling).
Firstly, opposite sides of the moon are reduced to the same scale, by adding 0.05 mag to the magnitudes (in all bands, B, V, VE1, VE2, IRCUT) on the Crisium side, and subtracting 0.15 mag from the Grimaldo side photometry. This brings both sides onto more or less the same sequence with airmass. We then estimate (by eye) the extinction as a function of airmass in each filter. The results are:
B 0.15 mag/airmass
V 0.10 mag/airmass
VE1 0.08 mag/airmass
VE2 0.06 mag/airmass
IRCUT 0.12 mag/airmass
Interestingly, the extinctions in IRCUT and VE1 are somewhat different, which we wouldn't expect as the filters are ostensibly rather similar. We should look into this further. In any case, these extinctions allow use to define lines about 0.3 mag wide around each sequence for each filter and pick off the best data. The result is this diagram:
A list of these frames is here:
There are 3162 frames on this list -- cut down from about 5000, so about 60% of the frames have survived the "tunneling" process. This is still a quite conservative list because I haven't
1) checked for the Lunar centering,
2) checked for "alpha", the scattered light parameter, except for those very obvious cases which showed up by eye.
Final plot: using just the good data, the extinction corrected
apparent magnitudes at MLO are compared to the JPL model as a function of Lunar phase. The scatter is small in some of the filters -- we are down to absolute photometry errors of just a ~ 0.1 mag relative to JPL, which is pretty good considering we cover a range of airmass mainly from 1 to 5, but with some of the data out to airmasses of 15!
There is a hint we can do better still, because there is still structure as a function of phase -- for example, the B band magnitudes at negative phase are a bit brighter still than the positive phases -- indicating we didn't get the correction for each side of the moon quite right. There are also trends in the reflected light with phase, but this probably reflects the fact that phase and airmass are somewhat correlated (small absolute phases tend to get observed at higher airmass, because the moon starts the night closer to the horizon).
From flux to AlbedoPosted by Daddy-o Nov 05, 2012 01:14PM
The earthshine comes from the various parts of the Earth that are turned towards the Moon, and the Sun - all the clouds and oceans and deserts and ice-caps that are illuminated and visible from the Moon contribute to the Earthshine. Which parts contribute most?
We take a representative image of Earth, as seen from space, and investigate where the flux mainly originates.
Splitting the above JPG image into the R, G and B channels we can analyses where e.g. 10, 50 and 90 % of the light comes from. That is - we seek the pixels that contribute these fractions of the total flux, and identify them in images. Note that R,G and B refers to other wavelength intervals than the B, V VE1 VE2 and IRCUT bands we have - our B band is bluer than the 'B' used in JPG images.In the three frames we see 3 rows (B, G and R, from the top) - on the left in each panel is the original R,G or B band image, while to the right are the pixels contributing to the 10, 50 and 90 percentiles of the total flux in the image. The order of the panels is: top left 90%, left bottom: 50% and top right is 10%.
In the 90% images at top left we note that the B image (top row) looks different from the R and G images - the light in the B band comes from atmospheric scattering - Rayleigh scattering, and aerosol scattering - as well as the ocean and the clouds; other bands have more of their flux coming from clouds.
Variations in the blue may therefore tell us more about the atmospheric state than do the other, redder, bands. The Rayleigh scattering is due to molecular scattering - as long as the composition of the atmosphere is the same this ought to be constant in time; but some of the blue scattering is also due to aerosols and thus we may have a tool to investigate variations in the aerosol load. The longer-wavelength bands will tell us more about the continents. All bands are quite dominated by clouds - a small cloud can reflect as much light as a larger un-varying continental area.
The above is repeated here on another image of the Earth - more realistic as it is half-Earth. Image from Apollo 8.
And here is the B,G,R images and the 90% percentiles:Top to Bottom: B,G,R, Left: R,B or G-band image - right: 90th percentile image.
We again see that the light contributing to the blue image (top) is more diffusely distributed than in e.g. the red (bottom) case where most of the light comes from variable features like clouds. This implies that we should expect larger variability in our albedo data for the red images than the blue image.
Data reduction issuesPosted by Peter Thejll Nov 04, 2012 01:29PM
While waiting for the CCD to be fixed, I am summarizing the discussions we have had on this blog. The material should be used largely in section 6 of the manuscript we submitted, leaving the other parts of that paper - or splitting it into tywo papers. Here is the submitted manuscript as well as the start of a summary the summary is really just a collection of links to this blog.
Real World ProblemsPosted by Daddy-o Oct 23, 2012 08:38AM
What is the effect on observational coverage if we have different numbers of observatories and observe in different ways?
Since this depends on what we mean by 'observational coverage' we define OC as 'largest fraction of the time with continuous observations'. Note that this is different from, say, 'largest fraction of days where at least one good observation was made', OK?
For 1,2,3 ... observatories chosen from the list of known observatories (in the IDL code observatory.pro) and evaluating at 15 minute intervals we get the following (non-optimal, but pretty good) results when January and July are combined:Upper panel: The red curve shows OC for Moon above 2 airmasses and Sun lower than 5 degrees under horizon. Blue is same but for 2 airmasses. The dip at 5 in the red curve is an artifact of the search method we use - exhaustive search between 44 available observatories would be too expensive so we seach for best of 100 random picks of 1,2,3,4 ... in the list of 44. Lower panel: The same, but evaluated for more stations, best of 200, and with a 10% random - uncorrelated - occurence rate of 'bad nights' (clouds, for instance).
We see that, compared to this
we have less OC - that's because that search was for 'Moon above horizon, Sun below' instead of the more realistic constraints used here.
We see that extending observations from AM 2 to AM 3 is equivalent to adding two observatories for midrange values.
We see that adding many more observatories is in the end a loosing proposition - the 7th observatory on the blue line adds nothing compared to the sixth.
a)More exhaustive searches can be made, but takes time. This would probably smooth the curves above and also uncover slightly better solutions.
b) We have restricted the site choices to the positions of known observatories. Since most observatories are on the NH summer months (when the Moon is not as high in the Northern sky) there is a handicap.
c) The method is slow - because the altitude of Sun and Moon are evaluated from very precise routines. Simpler and faster expressions for altitude could be used - but one for each observatory would be needed.
Real World ProblemsPosted by Peter Thejll Oct 19, 2012 10:58AM
The Moon is not observable all the time from a single observatory, all year round. To get complete - or almost-complete coverage we ask: where should the earthshine telescopes in the network be placed? The plot below shows, with coloured symbols, when the Moon is observable (defined as Moon up and Sun down), for 5 observatories around the world. MLO is Mauna Loa, MSO is Mount Stromlo in Australia, LCO is Las Campanas in Chile, SAAO is South African Astronomical Observatory. The underlying curve shows the earthshine intensity for one month. The Earth was modelled as a cloud-free sphere with bright continents and dark oceans - hence the zig-zag nature of the curve. Addition of randomly placed clouds would tend to dampen the amplitude of the zig-zags by about a factor of two.
When the underlying curve is not covered by coloured symbols it means that the Moon is not observable from any observatory. We note that this happens particularly when the earthshine is intense - that is near Full Earth, which is New Moon. This is clearly because the Moon is almost never in the sky alone when it is New (it has either risen shortly before the Sun, or will set soon after the Sun). Observability is best near Full Moon (New Earth) near day 15 - but then observing earthshine accurately is very difficult.
On average a single observatory experiences the Moon above the horizon and the Sun below, 25% of the time. Given the choice of these 5 observatories the total observability depends on time of year - given in the panels.
Clouds and the need for the Moon higher in the sky cuts the observability.
The above simulation is for two months of the year - the declination of the Moon changes with the seasons so the relative contribution by each observatory changes with time.
Data reduction issuesPosted by Daddy-o Sep 28, 2012 03:25PM
post, we noted that there is a daily progression in the obs/mod ratio of the DS/BS ratio itself (let's call this thing 'the relative albedo' from now on!). We speculated as to its origin.
We have now calculated the "relative albedo" with two sets of terrestrial models - one a uniform Lambert-reflectance sphere and one a non-uniform sphere with ocean-land contrast but with a Lambert reflectance, while keeping the set of observations constant. We compare these two sets of results:
The upper panel shows the "relative albedo" for a non-uniform Earth and the lower one for a uniform Earth. The linear regression slope is printed on each line.
We note that the 'progression' for V has changed sign in going from a non-uniform Earth to a uniform earth. The magnitude of the slope on B has changed by 50 %.
We conclude from this that the 'daily progression' is due to the terrestrial modelling of surface albedo on Earth.
Or: we can state that we are seeing actual surface albedo variations during a nightly sequence of observations!
During this set of observations (night of JD 2456046) The B-band albedo of Earth decreased slightly, while the V-band albedo rose. This is the albedo of the half-Earth shown in the figures in the post here
For fun, we plot here the B and V relative albedos for tow nights with overlapping phase intervals:
The two nights are 2456075 and 2456016. For the B progressions it looks like there is a difference, at phase -96 degrees of about 0.15 in the relative albedo, and given the scatter of the points we might be able to confidently see a difference 1/10th of that, I guesstimate. That is 1.5% at a given point in phase (that is, time).
BBSO discuss '1-2% per night'.
If we take the above results at face value we seem to be able to track albedo changes through a night. Averaging sucha progression, in order to get the 'nightly mean value' would, of course, yield an average value, and an average can be affixed with a 'standard deviation of the mean' which can become very small if you have enough points - and if it gives any meaning to average the points. It does not give that much meaning to average the above progressions since they change on scales longer than our observing sequences.
But anyway, we need to start understanding what BBSO does in their data-reduction. Notably - do they assume Earth is a Lambert sphere? If so, we have perhaps arrived at their "A*" quantity? If, even more, so - we now just need to observe for a few decades!
Given only 1.5 years of data we should for now focus on how we present the results, and how we get something 'meteorological' out of the data - as opposed to 'climatological'. For instance, does the level-difference between nights 2456016 and 2456075 correspond to a change in albedo in the relevant areas of Earth that we can determine from Earth observation data from satellites? More clouds, perhaps? 2456016 is the upper sequence.
Let's see! (more to follow...)
Data reduction issuesPosted by Daddy-o Sep 27, 2012 12:49PM
For all our observations, we plot the lunar phase (in the convention we use) against the azimuth (degrees East of North) of that observations.
We see that most positive phases are observations East of the meridian; most negative phases are West. Only rarely has the Moon been followed past the meridian.
Data reduction issuesPosted by Daddy-o Sep 27, 2012 11:38AM
We consider now the DS/BS 'ratio of ratios' - that is, the DS/BS ratio in observations divided by the DS/BS ratio in the model images. We have used the Clementine albedo map scaled to the Wildey levels, and Hapke 1963 reflectances in the model images.
We look at the data from several nights in the phase-interval from -130 to -50 degrees (Full Moon is at 0 degrees; negative phases implies
we are observing the Moon in the West (setting)):
What we see are short sequences of points - each sequence covering one night. We note how many of these nightly sequences 'dip' during the progression of one night. We note the smoothness of each nightly sequnce - this bodes well for our original idea that albedo could be extracted with high precision. Jumps from night to night are more troublesome! But we need to first understand the 'progression'.
Inspection of the time-stamps on the data in a 'progression' shows that time moves to the right - that is the 'ratio' plotted starts out high at a larger negative phase and ends as a lower ratio at a less negative phase (moves left to right). Since the observations are in the West we are looking at a setting Moon, or increasing airmass along a progression.
On a single night several things happen:
1) the Moon is seen through different airmasses, and
2) the Earth turns.
[recall that the synthetic models contain all the correct distances as functions of time, so effects due to that should divide out].
1) If the effect is due to the increasing (or decreasing) air mass we need to understand how a ratio of intensities taken from two areas on the Moon, obtained from the same image, can show a dependence on airmass. Differential extinction across the lunar disc? really? We are looking at 10-30% changes in a single progression!
2) Since these observations are all at negative phase they may all represent the same configuration of Earth and Moon - i.e. we could be seeing the effect of the same parts of the Earth rotating into view through the night. From a fixed observatory there is a tendency for the same areas on Earth to reflect light to the Moon on subsequent nights - in our case either the American sector and oceans on either side, or the Austral-Asian sector.
We should not forget 1 - but let us turn to 2 for now. We can extract 'scenarios' for the Earth at the moments of observation corresponding to the data-points above. Inspectingh the observing scenarios could perhaps teach us what is going on.
We select the progression near -91 degrees. That is a set of observations from JD 2456047.75 to 2456046.87, or about 3 hours (the date is April 30 2012). At the start the Earth was in this configuration (crosses on the globe show the part illuminated by the Sun):
While this is the configuration at the end islike this:
That is, the Earth has rotated slightly during the observations and
more of the Asian landmass has swung into view, meaning that more of the
earthshine is coming from the Asian landsmass, than at the start.
Can we estimate the expected change in Earth's reflectivity during this sequence? Yes. We have quite elaborate code for that, know as "Emma's code" due to a student who worked on it in Lund. It shows the Earth from any angle at any time, with superimposed albedo maps of clouds as well as continents. A reflectance law is imposed. I think there is even some Rayleigh scattering added to give the 'bright edge' of the Earth (see any Google Earth image at startup, as you zoom in - the edge is bright, and that may even be realistic).
That code is quite complex and is not run in a jiffy. [We need more students!] But we can make a perspective model of the Earth using simple IDL routines. We can wrap surface albedo and cloud maps onto a sphere and view it from the angle the Moon is at, for any given time. There is no reflectance imposed - just the albedo maps and the perspective view. The two situations above look like this with cloud maps for the moment of observations (taken from the NCEP reanalysis product TCDC):
The simple average of the brightness of the two images above are 42.6 for the former and 40.6 for the latter - so it is getting dimmer by about 4% as time passes - perhaps because bright Australia (cloud-free at this time) is entering dusk. The cloud map images are available at 6 hourly intervals only, so there has not been time for the cloud image to change - it is simply rotated and other bits have become hidden by the day-night sector advancing.
Post-Obs scattered-light rem.Posted by Daddy-o Sep 26, 2012 04:17PM
We have extracted the mean value of alfa (the parameter that describes how broad the PSF for a given image is) for a given night, and the corresponding value of the extinction coefficient for that night. We have only few nights where the extinction could be determined.
The plot of one vs the other looks like this:
(download and look, etc)
We see that there is a tendency for alfa to be narrowly constrained, and that the extinction has a broader distribution. In general there is no strong relationship between the values, but if we ignore outliers and the effect they have on the regression (plotted as a red line) we see a general tendency for high extinctions and small alfa values to be related: For B it is quite clear. V would be clear but for the outlier, IRCUT also seems to be clear. VE1 and VE2 are all over the place. A broad PSF is given by small values of alfa. We expect broad halos (i.e. broad PSFs) on hazy or turbid nights - nights on which extinction also should be large.
Factors that determine scatter in alfa are things like image focus and how the nonlinear fitting routine determined it should stop. Physical factors include haze and thin cloudyness on the night in question.
Factors influencing the scatter in extinction include the actual regression: we used all nights with more than 3 observations used to determine the airmass vs extinction line, and for which the determinations from 3 regression methods agreed to a S.D. of less than 0.02. The three methods were - ordinary least squares, and two 'robust' methods ("ladfit" and "robust_poly_fit" in IDL). While doing the actual regression it was necessary to eliminate some outliers by hand. Physical factors include haze and cludyness and whether the halo around the Moon was well captured inside the image frame.
The relatively low value of alfa for the VE2 filter is still not understood.
Real World ProblemsPosted by Chris Flynn Sep 24, 2012 12:09PM
We weren't sure of how exactly JPL models the Moon's brightness in HORIZONS, i.e.http://ssd.jpl.nasa.gov/horizons.cgi
The following plot shows that JPL calculates the actual observatory-object distance in giving the Moon absolute magnitude, but does not use any albedo map of the Lunar surface -- so it is symmetrical with phase on either side of new moon (say).
Lower panel : apparent magnitude of the Moon over a 12 month period (September 2010-2012) computed with Horizons, and shown in a narrow range of the illuminated fraction (40 to 50 percent). The scatter in the apparent magnitude around the trend is ~ 0.1 mag.
Most of this scatter is due to the changing distance of the moon around its orbit (between ~0.019 and 0.023 light minutes). The middle panel shows the magnitude if we correct the photometry to a standard distance (0.0215 light minutes) -- this reduces the scatter to 0.03 mag. (The distance provided by Horizons includes the position of the observatory on the Earth -- this can make a difference of up to ~12,000 km in the Moon-Observatory distance).
The upper panel shows the residuals in a least squares fit to the middle panel, as a function of Julian day over the 1 year sample period. This clearly shows that most of the 0.03 mag scatter in the middle panel is due to the changing Sun-Earth distance during the course of the year. Accounting for this reduces the scatter to <0.01 mag. We interpret this to mean that no surface features of the Moon are being included in the Horizons' apparent magnitude estimate, since we expect considerably more scatter than that (but we are working on this!).
Post-Obs scattered-light rem.Posted by Daddy-o Sep 20, 2012 11:41AM
At the moment we are displaying our results by providing plots of ratios of the DS/BS ratio in observations, to that in models. We do this chiefly to get rid of common factors - such as solar irradiance and distance-related geometry.
What remains in such a 'ratio of ratios' are the effects of:
1) The Earth's actual albedo,
2) the model's Earth albedo,
3) Earth's reflectance (real vs modelled),
4) lunar surface albedo's in the reference patches (real ratio vs model ratio)
5) effects due to the choice of lunar reflectance model.
We are really only interested in 1.
2 is an assumed value so that the results we get for terrestrial albedo are relative to that choice. We use a value of 0.31.
3 is a choice - we expect that any errors made in this choice will be seen as a phase-dependency in the results, and we can therefore control or at least understand it. Earth is more Lambertian than the Moon. The Earth has edge-darkening, the Moon has very little. We use a Lambertian model for an otherwise uniform Earth.
4 is observable, but only with difficulty - you need a good total lunar eclipse, then the albedos in the two patches on the Moon - or their ratio - can be measured. BBSO has done this. We have not (yet). In the model we make a choice, based on which lunar albedo map we use. As long as it is fixed the results will be relative to that choice. Perhaps we can use published images of total lunar eclipses to extract the ratio?
5 is a choice - we have models for Lambertian reflectance, as well as the Hapke 63 model and other, as yet untested, more advanced reflectance models. We expect that incompletenesses in these models will be seen as a phase-dependency in the results.
Of the above only some could possibly induce a phase dependency: 3 and 5.
We have reduced all data using both lunar models based on the Lamertian reflectance and the Hapke 63 model. We show them next - look here
for a discussion of what we are actually showing: 'ratio of ratios' and all that:
(as before, plot needs to be downloaded as it does not show up well on this blog).
The plot is for 5 filters with the EFM method applied. The left column is the ratio of DS/BS in obs to the same in model, while the right column is DS/total in obs relative to models, where 'total' means the disk-integrated brightness of all source counts (plus a few stars that we can ignore!).
The first page is for the Lambertian lunar reflectance. The last page is for the Hapke 63 reflectance.
We notice that left and right columns are quite similar. We notice that the 'jump up' at angles corresponding to about 40 degrees from New Moon is much smaller in the Hapke 63 model results. Inspection of the files that correspond to the individual points in the 'jump up' and those next to the jump, reveals that the 'jumped up' points have a different processing history: they are the result of coadding and averaging single image sequences, while the rest are stacks of observations that were averaged. Why these should be different is unclear, as yet.
We conclude that there was an effect of lunar reflectance model on our results - and that Hapke 63 is better than Lambert. This is not surprising as the Moon is well-known not to be Lambertian in its reflectance.
So - we are beginning to see observations constrain theory!
There is still some scatter to account for, and we shall return to this mattrer, using the estimates of nightly seeing available from the measured alfa.
There is also some 'slope' in the result wrt phase - so the EFM method has a success rate that is phase dependent. WHile the second-best method (BBSO logarithmic; not shown here) has some 'upturn' towards Full Moon (center of plot), the EFM has an even slope down towards FM. Perhaps some empirical fine-tuning of the method will remove this problem too.
Data reduction issuesPosted by Chris Flynn Sep 19, 2012 11:10AM
We have isolated the very best images in V band, in which the Moon is well centered and well exposed.
We measure the apparent magnitude of the moon in these images by simply measuring the total flux, and applying the standard photometry relations (i.e. http://earthshine.thejll.com/#post229
We correct for extinction of 0.10 mag/airmass (from http://earthshine.thejll.com/#post249
We then compare the apparent magnitude to the expected apparent V magnitude from the JPL ephemeris for the Moon (http://ssd.jpl.nasa.gov/horizons.cgi). This uses the relation quoted in Allen "Astrophysical Quantities", which actually comes from Eqn 8 of this paper: http://adsabs.harvard.edu/abs/1991PASP..103.1033K
The plot shows the difference in the apparent magnitude as a function of phase (new moon = 0). Blue and green show opposite sides of the moon.
There is a bit of scatter in these data, but there are two clear sequences around phases 50 to 100 showing that opposites sides of the moon differ in luminosity by about 0.1 mag. This is quite a lot less than we expected to see, viz. this post:http://earthshine.thejll.com/#post253
Post-Obs scattered-light rem.Posted by Daddy-o Sep 19, 2012 09:59AM
In a previous post (here
) we have compared the Wildey and Clementine albedo maps. These maps are important for our synthetic modelling code since the albedo (along with reflectance assumptions and correct geometries) are the basis of constructing realistic model images, used in analysis. We can compare these two maps very directly by accessing both and plotting the mean albedo in boxes at common lon,lat positions:
Evidently, the Clementine albedos are lower than the Wildey ones by a factor of two for dark areas and by some tens of percent at brighter areas.
This may be an explanation for the discrepancy we have between observed morning/evening brightness ratios and modelled ones, which Chris Flynn put his finger on.
We seem to have [not shown, but material could be inserted] an observed difference (expressed in magnitudes) of 0.12-0.14 magnitudes at absolute phase 90 degrees between the morning and evening integrated brightness. In models, based on Clementine albedo and either Hapke 63 or Lambert reflectances the difference is more like 0.3 magnitudes. If the Wildey map is more correct than Clementine, in terms of the highlands/mare albedo, then using the 'flatter' Wildey albedos in the synthetic code would help on the morning/evening brightness issue by lowering that ratio in the models.
The issue with using Wildey instead of Clementine is that Clementine is a global map - Wildey only covers something like -89 to 89 degrees in longitude and something similar in latitude so there is no remedy for modelling under lunar libration. We could 'scale' the Clementine map, using the above relation and see what that gets us, though.
Before we do that, we should fully understand how our own synthetic code uses the Clementine map - I believe there is more to its use than merely being used as a lookup-table. Hans will be able to tell us about this.
A second-order robust polynomial fit to the above data (where points where Clem > Wild have been omitted) is:
Post-Obs scattered-light rem.Posted by Daddy-o Sep 18, 2012 10:34AM
Earlier we showed
the performance of the linear and logarithmic BBSO method. Now we have added the EFM method.
As before, plots are not shown well on this blog, so please download this pdf file and look at it:
You see the same 5 panels on each page - one panel per filter. On the panels you see the behaviour of the 'ratio of ratios' against phase - that is, the DS/BS ratio in observations relative to a set of synthetic lunar model images.
First page: RAW data - i.e. no scattered light has been removed. Full Moon is at the centre of the plot so we see the effect of scattered light - the obs/model ratio is increasingly not 1 as we near FM.
Second page: the linear BBSO method has been appllied to images. We see a reduction in scattered light - points 'move down' towards the 1-line.
Third page: logarithmic version of BBSO method - a slight improvement is seen.
Fourth page - the new one: the EFM method - a quite large improvement is seen over the best of the others! Many of the selected points are now lying on a flat seqeunce, except points towards New Moon, and some outliers.
The EFM data has been 'selected' in the sense that the parameter alfa, determined from images, has been used to select 'good cleanups'. A histogram of the detected alfa values was made and a notable peak found and only images with alfa in a narrow range were used for the above plot. Alfa between 1.67 and 1.73 were picked. The absence of VE2 points is due to this - the peak of the VE2 alfa distribution is between 1.60 and 1.66. We must investigate whether these are 'good' solutions. [added later: a brief visual inspection seems to imply that the 1.7 solutions are the good ones - not the majority, which for VE2, lies in the peak at 1.63. Hmm.]
Speculatively, we note the 'turnup' of points towards New Moon. We ahave earlier discussed that this may be due to some feature or failure of the synethtic lunar image model to correctly portray the Moon at large phase angles. On the other hand the 'turnups' now look less like a gradual sequence, and more like a 'jump' up. The 'jumps' occur near 40 degrees from New Moon - .... is this the 'rainbow angle'? The jump seems large - the rainbow angle paper spoke not of factors of 2 and 3 and 4 but of percentages. On the other hand the phenomenon has never been observed, as far as we know. It would be nice to have points 'outside' the rainbow angle to see the jump go down again (if this is the rainbow, that is). We note here that as we get closer to New Moon the lunar sicle is narrow and it becomes increasingly important to place the 'patch' in which the BS brightness is measured - here is an opportunity to experiment with DS/total ratios, instead.
Work continues ...
Showcase images and animationsPosted by Daddy-o Sep 14, 2012 02:35PM
We have been using the Clementine mission lunar albedo map for our work. There is an older digital map, by Wildey (The Moon, vol, 16, 1977), which we obtained access to thanks to Tom Stone of the USGS.
Here is a graphical comparison of the two:
Here they are again, now interpolated to the same 2.5x2.5 degree grid so that they can be compared numerically, along with their relative difference:Upper left: Wildey map. Upper right: Clementine map. Lower left: the relative difference = (W-C)/C. There are some interpolation artefacts on the rhs of both maps as well as in some of the edges, brought about when the full-globe result from Clementine (orbiting lunar satellite) and one-side-only result from Wildey (ground-based telescope observations) are accessed.
The map of the relative differences reveals an overall albedo offset with Wildey being darker than Clementine. In the Mare areas differences of 40-50% are found whil ethe brighter highlands have differences in the 5-20% range. The differences in albedo is therefore not a constant but depends on albedo.
We see no obvious longitude or latitude dependence in the difference. This may be relevant to whether we are interpreting the Clementine map correctly in terms of angle-dependence in the conversion from observations to the map. Or the same convention was used in the Wildey map, produced 30 years earlier!
[added later: more here
Post-Obs scattered-light rem.Posted by Daddy-o Sep 13, 2012 11:05AM
We evaluate the effect of two different scattered-light removing techniques on our data, comparing to no removal at all. We do this by considering the DS/BS ratio - and we look at the ratio of this ratio in observations to the same ratio in models - so it is a ratio of ratios, ok?
The graphics do not reproduce well on this blog so I put a link to a pdf file here: download it and read on.
There are 3 pages to look at, and on each page there are 5 panels - one panel per filter, one page per data set.
The first data set shows the ratio of ratios extracted from data where only a bias has been subtracted, plotted against lunar phase. Second page shows the ratio of ratios when the LINEAR BBSO method has been applied, and the last page shows the ratio of ratios when our logarithmic variant of the BBSO method has been applied.
Full moon is at phase 0.
We see that there is a scatter and there is a systematic dependence on lunar phase - the points 'curve up' towards full moon (the middle) and towards the edges (new moon).
Near Full Moon it becomes increasingly difficult to remove scattered light because the BS is closer and closer to the patch on the DS where photometry was extracted. The Eartshine is also weaker and weaker as you approach Full Moon because that corresponds to approaching New Earth.
Therefore a hypothesis was that the 'curve up' towards Full Moon was due to incompletely removed scattered light. Looking at the raw vs linear vs log methods it is evident that this 'curve up' is substantially reduced by application of either of the scattered-light removal methods. The linear method does well and the log method adds an increment of improvement.
I believe these plots are a demonstration that BBSO linear is not perfect - and that small improvements are possible. It is open to discussion what BBSO has done about this problem - light seems not to be completely removed in their method. How do they average their data to compensate? We should note that our image scales and observing methods (ND filters; coadd etc) are not identical.
Soon we will add a third method - the EFM!
Data reduction issuesPosted by Chris Flynn Sep 11, 2012 05:22AM
The plot shows the apparent magnitude of the moon in V and B as a function of lunar phase (phase=0 is new moon).
We measured the flux in images in which the filter was reliably V or B and used the transformations determined from NGC6633 (i.e. http://earthshine.thejll.com/#post229
) to get the apparent magnitude.
These data have been obtained for a large range of airmass (z) -- from z = 1 to 10, with most of the data in the range z = 1 to 3. We derived extinctions of 0.10 mag/airmass for V and 0.17 mag/airmass for B, by comparing to the apparent V magnitude from the JPL ephemeris for the Moon (http://ssd.jpl.nasa.gov/horizons.cgi) (more about this below). The solid black line shows the V apparent magnitude as a function of
phase after extinction correction, and adjusting the zeropoint by 0.2
mag in V to fit.
Note that B is ~ 1.0 mag fainter (i.e. B-V ~ 1.0, as we'd expect).
The airmass fits are shown above : the plot is the difference between the apparent magnitude from the JPF ephermeris and our transformed instrumental V band (or B band) magnitudes, shown as a function of airmass. The two lines show 0.10 mag/airmass (V band) and 0.17 mag/airmass (B band) -- they are not fits. There are some bad outliers, especially in the V band, which are probably due to the incorrect filter being in the beam.
Post-Obs scattered-light rem.Posted by Daddy-o Sep 07, 2012 09:12AM
By running the computers for one week we have been able to process all observed images (both singles and those that had been coadded from stacks) with the EFM method, and thus estimated the exponent, alfa, that the basic PSF must be raised to to model the halo around the Moon. Plotting the histogram of 9000 values for alfa we get:
We see a broad distribution from about 1.3 to 2.1. Inspection of the results reveals that only the images with alfa values very close to ~1.7 are any good. The lower values correspond to hazy or even partially cloudy nights. The values higher than ~1.7 have yet to be examined.
A typical (histogram equalized) image of the residuals for one of the images with alfa near 1.71 is here:
A 'slice' across the image shows this:Upper panel is the slice - black being the image, red being the fitted halo; second panel is a detail view of panel one showing the DS, vertical dashed lines showing the limits to the sky on which the halo is fitted; third panel shows the difference between the black and red curves in panel two.
We see that the EFM method has been able to fit the sky so that it is essentially just noise - even on the BS (right) side of the disc, and that the DS has been revealed for a wide area onto the disc itself, only near the terminator is there a problem with the subtraction - the residual dips down.
Data reduction issuesPosted by Daddy-o Sep 05, 2012 09:09AM
As discussed here
, the CCD camera became twisted in its thread on the telescope at one point. The problem was fixed, but this means that some of our images have a slight rotation about the frame mid-point. This influences the success of subseqeunt data reduction steps: especially the steps that depend on extracting flux from specific areas on the lunar surface.
We therefore tested for the presence of a rotation angle by correlating a synthetic image made for the observing moment with each and every observed image, rotating the synthetic image until the correlation was maximum - in 1 degree steps.
We plot the detected best rotation angle as a function of image sequence number and date:Top frame: detected image rotation angle vs sequence number, Bottom panel: angle vs observing day since start.
It certainly seems that almost all images up to number 1800 or so has a rotation angle of some -7 to -8 degrees. That seems to correspond to just a few nights near night 40-50. The detection of rotation is a bit spotty so there are also other episodes where a rotation angle other than 0 is detected - such as images 2000-2500. That more intermittent episode corresponds to a few nights near night 180, but there are a few more examples near night 220.
The CCD twist was correctd by Ben on JD2455991, and this datum is shown as a vertical dashed line in the plot above. Since this is not consistent with the angles measured we have to say that the test so far has been inconclusive!Added later:
Actually, it was not impossible to inspect the relationship between model images and observed images visually and to confirm when an obvious image rotation was present. Partial results (note: more points than above) look like this:
Here color coding indicates in red
the images that so far obviously have a rotation problem, and in blue
images that show no obvious problem.
The presence of blue symbols at large rotation angle must be due to failure of the algorithm for detecting rotation! It is not an easy problem to solve - at New Moon there is precious little to correlate images on, unless the DS is used - but the presence of the halo gives problems, so that histogram equalization is not an obvious remedy.
A fixed derotation for the detected nights could be implemented - this affects some of the early observing nights where single images were taken (not stacks).Added even later:
By manual inspection and image comparison, the following de-rotation angles were found for the JD in the beginning of our sequence:
Apart from 2455859 all angles before 2455886 were clear to find. 2455859 was hard to inspect as it is very near New Moon and almost no features were detectable.
As a working hypothesis, let us assume that all images before 2455886 must be rotated by something like 6 or 7 degrees, to bring them into good alignment with their synthetic models.
Post-Obs scattered-light rem.Posted by Daddy-o Sep 03, 2012 03:26PM
The results for applying the linear BBSO scattered-light removal method to all data are further considered. Here we show data where the lunar disk is well centred and the radii extracted are between 131 and 149 (real range is 132-148) pixels. (Sorry about the image rotation!):Rotate image and then use this caption: Results from comparing model images to observed images - models calculated for the observing moment but no halo generated. In the observations the halo has been attempted removed via the BBSO linear method. Column (1) DS counts to total counts for observation (black symbols) and same for Model in red symbols, against lunar phase; Column (2) observed DS to total ratio against airmass; (3) ratio of the DS/tot ratio for observation and models.Note that Full Moon is at 0 degrees phase.
We note that for lunar phases between New Moon and about 100 or 110 degrees the model and observed ratio of DS to total flux behave similarly. For phases closer to Full Moon the model DS/tot ratio is much smaller.
That is probably because near Full Moon the halo gets closer to the DS patch where the DS is measured: In the model no halo is present so the DS brightness is not polluted until the patch is actually part of the BS - in the observations the halo moves with the lunar terminator towards the BS and becomes harder and harder to remove so that it starts to pollute the DS - even if measured in images with some of the halo removed.
In the middle column we see a fair spread in observed DS/tot vs airmass. Most of the observations are for AM < 3, while a few have even higher airmasses. In the bulk of the observations we see no filter-to-filter consistent evolution of DS/tot ratio against airmass, and conclude that there appears to be no dependence on airmass; this is good as the whole point of simultaneous observing of DS and BS is that external factors, like extinction, affect both sides identically.
In the last column we see the ratio of the DS/tot number between observations and models. The behaviour noted from the first column, above, is evident here - the DS/tot ratio in models is much smaller than in observations for phases inside 110 degrees or so and the ratio of ratios goes up accordingly. For phases between 110 or 100 degrees and New Moon we see a more constant behaviour consistent with the observation made above that model and observations behave similarly. We see, however, a consistent pattern from filter to filter - there is some 'curvature' with phase. The inner part of this could be due to lingering halo effects, but the outer 'curve up' must be due to effects in the model, not in the observations. Near New Moon it seems that the DS/tot ratio for observations grows compared to the same ratio for models - the models' DS is not as bright as in the observations near New Moon. We note that the DS model brightness is a function both of the reflectance description of the Moon and of the Earth.
1) There is indication that halo-removal near Full Moon (phases between 0 and +/- 100 degrees) is increasingly incomplete.
2) There seems to be little or no effect of airmass on the DS/tot data.
3) There is some indication of a reflectance problem in the models for phases approaching New Moon - either in the Earth description or in the Moon.
1) is good news - we thought we could only do removal up toi half Moon - the limit seem sto be 10 or 20 degrees beyond that, towards a fuller Moon. 2) is very good news - it opens up for our method to be applied at low altitudes, i.e. for small lunar phases near New Moon.
3 should be investigated by now redoing the whole reduction but with some simple and single change in the modelling - we cannot change the BRDF for Earth (it is always Lambertian), but we can change the lunar BRDF from Hapke 63 to Lambertian.
In these models the Earth was modelled as a 'cloud free Lambert sphere' with ocean/land contrast built into the single-scattering albedo. The Moon was modelled by the Clementine single-scattering albedo map with Hapke 1963 reflectance.
Post-Obs scattered-light rem.Posted by Peter Thejll Sep 01, 2012 06:15PMResiduals image of Moon minus model of BS and its halo, fitted to the sky (i.e. the EFM method). The green and blue colors are a few tenths on both sides of 0. The mask used covers the image except the disk of the Moon.
In this application of the EFM to an observed image of high quality we see that the residual sky is not even. Particularly on the BS (to the right) there are some streaks. A blob at the bottom also shows unevenness, as does the striations to the left of the DS.
This illustrates a shortcoming of the EFM. Since the PSF used is rotationally symmetric it is possible that the unevenness seen above are in the observed image - perhaps some sort of reflection effects in the optics. The EFM is a global method seeking to subtract all the scattered light with one model. The BBSO method is essentially local - it estimates the scattered light in a wedge off the disk. On the other hand the BBSO linear method almost certainly removes too little of the halo as you get nearer to the BS. The EFM method attempts to model all parts of the halo.
Data reduction issuesPosted by Daddy-o Aug 31, 2012 02:00PM
Here are some of the first results from applying the BBSO linear method - the reductions are slow so more will be on hand later! Sorry about the low quality in the image - there must be a way to do it better, but ...
Rows 1-5 give results for each filter. Ignore columns 1,2 and 3 for now - they are diagnostic. Column 4 shows the ratio of observed terrestrial albedo to modelled terrestrial albedo as a function of lunar phase.
We pick out the DS as either Grimaldi or Crisium depending on which is in the DS. We then calculate the observed DS brightness divided by the brightness of the whole disk. We then do the same for the synthetic model, and plot the ratio of the two.
This may seem a strange quantity to plot, but consider that in the (unlickely case) that we both had perfect observations and the model was correct in all aspects, then we would see a ratio of 1.0.
If the model is somehow wrong - for instance if the phase function it is based on is unrealistic then the ratio would have a phase dependence.
As it is, we do not have perfect observations and we see a fair bit of scatter. The scatter comes about for several reasons - first of all the observations have Poisson noise - we are extracting a small 4x4 pixel patch on what is the dark side where counts are probably on the average of 5 or so. Additionally we have noise from the alignment between the actual Moon and the coordinates we have calculated from which to extract information - there is a missmatch of up to several pixels here, so for a small area like Grimaldi a few pixels error in placement brings you into the bright surrounding areas. For Crisium, which is larger, this is less of a problem.
Finally there is a still un-solved problem with synthetic models and observations apparently being off by some small amount in terms of a small rotation. This may be from the days when the CCD camera was actually physically twisted by a few degrees in its placement on the telescope. In such cases the intensities of the pixels extracted in the synethtic model and observed image are even more different.
So, some things to work on are:
1) Use larger DS patches, so that the Poisson statistics are not as much of an issue.
2) Put the patches in uniform areas on the Moon so that missalignments do not cause acquisition of contrasting areas. Inside large, even Mares or on the brighter highlands.
3) Use better estimates of disc centre and radius.
4) Figure out a way of aligning the synthetic model and the observed image better.
PS: More data are available all the time so the figure above will update now and then.
Have implemented 1 and 2, by enlarging the area inside Crisium that is used, and using a rectangle in the highlands south of Grimaldi instead of Grimaldi itself. Also trying 3.
Post-Obs scattered-light rem.Posted by Daddy-o Aug 30, 2012 12:50PM
The linear BBSO (i.e. as done by BBSO) method and the modification that operates on logarithmed images can be compared:A strip, 20 rows wide, across the image was made and the rows of that strip averaged, and the average plotted. Black is the original bias-subtracted image, red is theordinary linear BBSO method and blue is the log-method. We plot the absolute value of the fluxes to avoid problems with log.
Both the linear and the log method does well on the sky at left, but there are differences on the edge-near disk. Tests on synthetic images, where we know the flux to expect on the disk, has shown that the log-method is more accurate than the linear method.
The residual mean (about 2) is unsettling, but has to do with the use of absolute values and the averaging over several rows - inspection of the images reveal values distributed around 0.
Data reduction issuesPosted by Daddy-o Aug 29, 2012 03:16PM
We have to determine the radius and centre of the lunar disc in order to reduce observations.
In doing that we must be aware that differential refraction causes the Moon to appear non-circular as it comes closer to the horizon. Using a formula from the Nautical Almanac Explanatory Supplement we generate the following table for 600 mmHg, 10 degrees C and 30% relative humidity:
Z d_refr am
27 1" 1.1
69 6" 2.8
75 12" 3.9
where z is the zenith distance in degrees, d_refr is the differential refraction in arc seconds and am is the airmass. The differential refraction is calculated over a 1-degree distance centred on the given zenith distance. Our FOV is about 1 degree wide and one pixel covers about 7".
We thus see that the Moon is differentially refracted by less than one pixel up to about 2.6 airmasses. Two pixels are reached about 26 degrees (4 airmasses) above the horizon.
Some of our observations are certainly close to the horizon, as we have tried to observe when the lunar phase is near or less than 30 degrees (at Newish Moon).
At 30 degrees the uncertainty in our determination of disc radius and disc centre (based on a circular assumption) is thus starting to be challenged by the differential refraction.
Data reduction issuesPosted by Daddy-o Aug 29, 2012 02:14PM
In order to extract fluxes from particular areas of the Moon we need to take lunar libration into account. Based on the synthetic model code Hans wrote we can do that now.
On the left is the synthetic model. On the right is the observed image. On it are some dots - they are supposed to be inside Mare Crisium and Mare Nectaris and two other locations on the darkened side. We see that we nail Nectaris and Crisium. Doing this for every observed image we will be able to extract feature fluxes and compile DS/BS statistics.
Data reduction issuesPosted by Daddy-o Aug 27, 2012 04:22PM
Since the setting of the colour filter (as well as the shutter) was unreliable we must find a way to detect which images are taken through which filters.
Here we plot the raw fluxes (counts divided by nominal exposure time):Total lunar Fluxes plotted as magnitudes against the lunar phase (New Moon is at 0). Bright is up, faint is down.
The data for each image (black symbols), here expressed in magnitudes, are overplotted with a phase law (red) inspired by that in Allen "Astrophysical Quantities", except we modify the coefficients in that and use instead:
Notably the coefficient on the linear term is about half of what Allen specifies.
Particularly in VE1 we note the presence of two sequences of data. We have 'fitted' (by eye) the sequence that is represented on both sides of the new moon to the Allen phase law. The orphan sequence is below the adopted data suggesting that a filter with less transmission was obtained here when VE1 was requested.
For B it seems that intermittency causes some fluxes to be higher, but they do not fall in a delineated sequence so cannot be identified with a filter.
The V filter seems to have the same problem, although less so.
VE2 is also somewhat 'broad' in its distribution.
IRCUT shows two sequences.
The VE1 and IRCUT filters are extremely similar in transmission properties and are quite similar in the plot above, including the presence of the 'second sequence'.
The 'second sequence' is quite similar in flux to the V observations, and it is consistent to say that when IRCUT and VE1 failed to be chosen V was obtained instead.
This seems to also apply to some of the outliers in VE2 and B.
We thus suggest that a working hypothesis for the failure of the filter wheel is that when it failed the V filter was selected instead.
We next proceed to eliminate the outliers in each filter so that a dataset can be defined which will allow identification of the extinction laws in each filter. With that in hand it may be possible to 'tighten up' the data and move towards a 'golden dataset' from which also DS fluxes are worth extracting.
The presence mainly on the left side of the diagrams of a 'second sequence' of data implies that something like a mechanical problem is behind the FW failure - because the telescope is flipped over the meridian to observe mainly in the East or the West depending on whether the Moon is rising or setting (before or after New Moon).
Post-Obs scattered-light rem.Posted by Daddy-o Aug 27, 2012 11:36AM
In order to precisely remove scattered light from the observed images we need to know the centre of the lunar disc in image coordinates, as well as the radius. These numbers are used by the BBSO method, while the EFM method can work without them. Extraction of fluxes from designated areas of the Moon also requires knowledge of the disc coordinates.
We have, as described elsewhere, found a fairly good way to estimate disc centre and radius, and have more than 5000 images (singles or sums of stacks) with know coordinate estimates.
We can check on the quality of these data by inspecting the time evolution of the disc radius in terms of the lunar synodic period (27.322 days).
Plotting the detected radius against observing time modolu 27.322 we get, for the 5 filters:
There seem to be some outlier groups, as well as a general scatter. The scatter is on the order of 2-3 pixels while the outliers reach 5. These outliers can be identified and the relevant images inspected.
We fit a general sine curve to the data, and get:
Offset Amplitude Period
141.169 7.98152 27.6367
+/- 0.0108026 0.0166443 0.00208776
140.663 7.81091 27.5796
0.0122441 0.0199136 0.00187041
140.647 -7.47896 27.5736
0.0413606 0.0837630 0.0111588
139.845 8.51341 27.5771
0.0245651 0.0383531 0.00435187
140.744 7.98575 27.6255
0.0358718 0.0596396 0.00708377
The period is not close to the expected 27.322 days. We expect this is due to a poor fit (in turn due to the outliers). We identify the outliers. 108 images are found that have radius more than +/-3 from the fitted sine curve.
Upon inspection, it turns out that not many of the identified outliers are obvious 'bad images'. The determination of radius and disc centre is therefore somewhat deficient.
Control SoftwarePosted by Peter Thejll Aug 25, 2012 08:50PM
Following the power surge and the consequent failure of the system, I installed SOLIS in case the software and drivers had been hurt by the power surge. This was apparently not a good idea - during construction of the system it was realized that 'drivers' from SOLIS and 'drivers' from the SDK are not the same and that only drivers from the latter must be present.
I therefore today de-installed (using the Windows 7 Control Panel) the Andor SOLIS package - as well as the unwisely installed Andor SDK package (both, apparently, with their provided drivers).
I then tried to run the 'setup.exe' present in the Andor SDK zip file - the setup offered to install itself into
C:\Program Files\ANDOR SOLIS\Drivers
which is supposedly not right - the SDK should install into
C:\Program Files\National Instruments\LabVIEW 2009\user.lib\Andor Tools
according to Ahmad. So I rebooted the system. Actually, it seems I chose 'shutdown' so it is not coming back up - must ask Ben to physically reboot.
Error budgetPosted by Peter Thejll Aug 21, 2012 12:33PM
A required image analysis step is the determination of lunar disc centre and radius.
We currently use a hybrid method: First we fit circles to points on the BS rim (found as an image by using edge-detection image-analysis methods: SOBEL filters!) of the lunar disc image - this is done many times using different points and then the median is extracted for x0,y0 and radius. These values are then used as starting guesses for a more refined method that searches a range of possible values near the starting guesses and determines a 'winner' based on how well a synthetic circle matches the 'rim image' generated above.
We compare the statistics of the starting guesses and the final adopted values. In two separate runs on 88 different images we assemble these values:
x0 start guess: 311.82352 +/- 2.7447506
x0 final guess: 310.44694 +/- 2.3127377
y0 start guess: 248.49938 +/- 1.1339389
y0 final guess: 248.70556 +/- 1.0812740
radius start guess: 136.32425 +/- 0.98834739
radius final guess: 136.90833 +/- 0.44461002
x0 start guess: 311.78076 +/- 2.9802184
x0 final guess: 310.46767 +/- 2.3079720
y0 start guess: 248.51968 +/- 1.0747726
y0 final guess: 248.70327 +/- 1.1017349
radius start guess: 136.42162 +/- 1.2634192
radius final guess: 136.90870 +/- 0.40265642
x0 is found slightly more precisely with the final method (2.3 pixels vs 2.8).
y0 is found with similar precision in the two methods (about 1.1 pixels).
radius is found better with the final method (0.4 vs about 1 pixels).
We seem to be able to determine the centre of the lunar disc with image analysis techniques to the 1-2 pixel level, and radius to better than half a pixel.
Error budgetPosted by Chris Flynn Aug 21, 2012 11:53AM
We have looked at the error in measuring the flux in a patch on the earthshine side if the position of the patch is uncertain by a few pixels.
A circular patch (aperture) was chosen, as shown by the green circle below:
The aperture is 31 pixels in radius, and is in a not particularly uniform luminosity area of the lunar disc.
The amount of scattered light into this area is rather small (~<10% of the flux) and has been ignored.
The flux in this aperture was computed for the correct registration (no offset in x or y) and for offsets of 1 and 2 pixels in both x and y (i.e. a rectangular grid of -2 to 2 pixels offset in x and the same in y).
Simply summing the flux in this aperture yields a 1.5% error in the flux, if the registration error is up to 2 pixels. The error reduces to 1.0%, if the registration error is just 1 pixel.
We tried rolling off the edge of the aperture with a cosine function (i.e. from 1.0 to 0.0) -- it starts 6 pixels inside the edge of the aperture. We weight the flux in each pixel by this function. Pixels inside this rolloff zone are weighted with 1.0.
The improvement is rather small. For this "soft tophat" aperture, registration errors of order 2 pixels lead to a flux error of ~1.4%. This is to be compared to the error of 1.5% for the hard edged aperture. If we can achieve registration errors of 1 pixel, the soft edged aperture yields a flux error of 0.8%, compared to 1.0% for a hard edged aperture.
Next we tried a much more uniformly illuminated region of the moon:
The results are much
better. In this region, a registration error of up to 2 pixels yields flux errors in the patch of order 0.2% (31 pixel hard edge aperture) and 0.15% (31 pixel soft edge aperture). This is very acceptable!Conclusions:
Using a soft edged aperture helps ameliorate registration errors, but not as much as we had thought.
Selection of uniformly illuminated patches helps much more! Doing both is a good thing.
Real World ProblemsPosted by Daddy-o Aug 13, 2012 03:06PM
The image below is taken at the IIWI computer with the CCD camera
attached to the board, sitting in a PCI slot on the IIWI. The software
used is SOLIS.
This image looks like things we have seen before. The broad stripes are reversed but that may be something in the display software settings. It is a very primitive setup - all I have managed to do is take a
picture at some long exposure time. The CCD is still attached to the
telescope, I think, so the shutters are closed and what we see is a flat+bias
frame. As it was dark in the dome when the image was taken the 'signal' is dark current - not light. The bands are structures in the flat
field. The spots are noise. The large spots are possibly CR hits?
I think the image proves that the camera is able to take pictures. The
noise may be due to a damaged cooler or - more probably - that the
cooler is not switched on (I don't know how to do that) yet.
It is at least an image from the camera - so board and camera must be
OKish. Why then are there no images when the camera is attached to the PXI? One
answer could be that the PXI was damaged during the MLO power surge. I
am leaning towards that theory now.
Added later: Here is an image with cooler ON.
The minimum values is 402 - a bit high, but at least it now looks like a bias frame. I would say that there is nothing wrong with the camera or its board. So the problem must be in the PXI!