Exploring the PSFPosted by Daddy-o Oct 07, 2013 02:22PM
In this entry: http://iloapp.thejll.com/blog/earthshine?Home&post=374
we considered a blind deconvolution method.
The author, Michael Hirsch has kindly been in touch with us and pointed out that the isoplanatic patch is small so using the OBD on instantaneous images from a typical stack, will yield speckle-like PSFs that are then smeared by the method. A smeared PSF will cause the concurrently deconvolved image to have too sharp edges! This is what we see.
We should instead either
a) use subimages the size of the isoplanatic patch and receive the PSF for that image and a deconvolved version of that image, or
b) generate a series of stack-average images and apply the OBD to that, receiving the time-average PSF and the deconvolved image.
Of course, we thought we were doing b), but should instead perform the time averaging before OBDing.
Doing a) would not solve our halo-problem as the isoplanatic patch is small and we are looking for info on 1x1 degree PSFs.
Testing to follow ...
Note that night JD 2456076 is the one with most (15) 100-image stacks in all filters. You could get 30 in a row if you combined IRCUT and VE1 ...
Exploring the PSFPosted by Daddy-o Oct 04, 2013 12:48PM
By careful analysis of the halo around stars, Jupiter and the Moon, we have arrived at an understanding that the PSF is related to a power-function 1/r^alfa, where alfa is some number below 3.
It is also possible to estimnate the PSF using deconvolution techniques. In particular, the 'multiframe blind deconvolution' method produces estimates of both the deconvolved image and the PSF, without any other input than the requirement that the output image and the PSF be positive everywhere.
There is matlab software provided here: http://pixel.kyb.tuebingen.mpg.de/obd/
and a paper here: http://www.aanda.org/articles/aa/pdf/2011/07/aa13955-09.pdf
Fredrik Boberg and I - well, Fredrik - has tested the software on images pulled from a 100-stack on (nonaligned, but almost aligned) images. The output comes in the form of estimates of the PSF and estimates of the deconvolved image. The deconvolved image has an unnaturally sharp limb - but more on that somewhere else - for now, let us see the PSF!
The estimated PSF is highly non-rotationally symmetric, but does have a central peak. The 'skirt' wavers up and down over many orders of magnitude. A radial plot of the PSF and a surface plut looks like this:
The red line shows the 1/r³ profile. The elliptical shape of the PSF causes the broad tunnel seen above - the divergence at radii from 30 pixels and out is the 'wavering skirt'.
The PSF does appear to generally be a 1/r³ power law, which is encouraging. It appears to be quite flat out to 2 or 3 pixels - which also generally matches what we have seen in the PSFs empirically generated from imags of stars.
Test on a synthetic image convolved by a known PSF.
pre-align the images in the stack - perhaps a slow drift causes the non-round psf. Repeat the above for several stack from different nights.
Repeat the above for stacks taken through different filters - any indication that PSFs are different?
From flux to AlbedoPosted by Daddy-o Sep 26, 2013 08:56AM
Since we are calculating absolute calibrated B and V magnitudes (on the 'lucky night' 2455945) for the DS and the BS we can convert these to surface brightnesses, for comparison with e.g. Pallé et al published work.
The formula for surface brightness is
mu = mag + 2.5*alog10(w*w*N)
where mag is the magnitude determined from a patch containing N pixels, with each pixel covering wxw arc seconds. In our case
w=6.67 arseconds/pixel and
N is 101 and 113 for the 6x6 selenographic degree patches we use (+/-3 deg). With the magnitudes for B and V, BS and DS from the paper - but using N=1, since we report average magnitudes per pixel, we get:
mu_B = 14.29 m/asec²; SD=0.06
mu_V = 13.54 m/asec²; SD=0.06
mu_B = 6.21 m/asec²; SD=0.05
mu_V = 5.31 m/asec²; SD=0.06 (all SDs are internal error estimates based on pixel bootstrapping with replacement).
BS is about 8.23 magnitudes brighter than the DS - there are published numbers for these quantities (e.g. Franklin). His Table 1 has differences of about 10 mags between DS and BS. He may be talking about magnitudes per area - not magnitudes per pixel, like we are.
Pallé et al in this paperhttp://adsabs.harvard.edu/abs/2007AJ....134.1145M
give plots showing mags/asec² and for the phase we have (about -140 on their plots) they have a BS-DS difference of 8.4ish mags - so we are within 0.2 mags which seems possible, given the scatter they show in Fig 1.
Not sure I like or understand why Franklin is 1.5-2 mags different in the DIFFERENCE - would that come about when you differ by mags/pixel and mags/area?
From flux to AlbedoPosted by Daddy-o Sep 18, 2013 03:21PM
In this entry
we introduced a Monte-Carlo based model for the spectrum of earthlight. We are able to use the model to estimate B-V for outgoing light, given input for cloud albedo, surface albedo and so on - and cloud fraction.
From the NCEP reanalysis project we can take the global total cloud fraction product ('tcdc') and calculate the global mean value for different times (we pick year 2011 here) and use the values for the B-V model.
I have done this. In 2011 TCDC varied between 50 and 55 percentage points. delta(B-V) [i.e. the differences between the solar spectrum B-V and the earthlight B-V, in the model] varied from 0.090 to 0.075 at the same time. That is, earthlight became more solar coloured for larger cloud cover - good - and implies that Earth is bluer when there are fewer clouds.
The change in B-V is 0.015 mags - can we measure that at all?
In our little paper we have relevant results. Uncertainties on B and V are at the 0.005 to 0.009 mags level, so that differences are at the 0.012 level (worst case) for a B-V value - on the 'lucky night'.
But on the Full Moon night of 2455814 we have errors of just 0.001 and 0.002 on B-V.
We need to understand why the error can be so different. We promised 0.1% accuracies at the start of this project - and seem to be able to get it with 0.001 mags error - but why not on both nights?
Obviously, the error on the DS will be larger since there are much fewer photons - but this does not help explain why the errors are similar on the 'lucky night'.
One point is that on 2455814 we measure B-V from one image generated by alignment of the B and V images in question. On 2455945 we measure B and V in seperate images and take the difference of those means. There are 'cancellation' issues at play here - surface structure will add to the variance of single -band images while some of the structures cancel (particularly if the images are well centred) in difference images.
Added later: yes, there is an effect (obviously, duuuh) - if we calculate the DS-BS difference from areas in the B-V image, instead of in the B and V images seperately, and perform bootstrap sampling on the pixels involved, we get the mean over the resamplings and its standard deviation to be
-0.155 +/- 0.005
whereas the difference between <B> and <V>, and its error calculated from error propagation, is
-0.154 +/- 0.014.
That is, we have a third the uncertainty. This will be used in the paper.
The uncertainty is still a bit high, though. (5 times what we promised, or to be fair about 5/sqrt(2)=3.5 times. since the above is a difference and not a directly measured quantity - which was the thing about which we made promises!).
It could be that noise on the low-flux DS is dominating here - this remains to be seen. And we still need to understand why it is still higher than for the Full Moon night - but things are making a bit more sense now.
Links to sites and softwarePosted by Daddy-o Sep 10, 2013 07:50AM
Here is a link
to a site that lists transmission curves of many color filters - e.g. Wratten filters.
like Schott RG665 is most like VE2, and might be the long-pass filter we need on a modified SIgma SD100 camera. We need it with 58mm thread, though.
Wratten #25A is available from B&H with 58mm thread but has its cutoff at 620 nm.
B&H have a 58mm Polaroid that looks
ok. Transmits from 720 nm and up.
From flux to AlbedoPosted by Daddy-o Sep 05, 2013 12:48PM
We have noted interesting behaviour of our data at the lunar limb, link here
. It seems that our data show a convergence of intensity ratios, between certain symmetric points on and near the limb, towards unity. We saw the same in model data.
The model we use is based on the Hapke 1963 BRDF (i.e. reflectance) model and looks like this:
BRDF ~B(phase)*S(phase)* 1/(1+cos(e)/cos(i))
are the angles of emission and incidence. For a given instant the phase is the same everywhere (almost) so the only angular dependency that remains is the last term above. On the limb we can have various values of i
but only one value of e
- namely pi/2 - recall that i
is the angle between local normal vector and the Sun, while e
is the angle between local normal vector and the direction towards the observer.
cos(pi/2) is 0, of course, so the last term above is unity everywhere on the lunar limb. The ratio of the reflectance in two points on the limb therefore reduces to unity. The ratio of two limb intensities
should therefore be equal to the ratio of the albedos
at those two points.
So, there is no clue here as to why we observe, and the model gives, intensity ratios near unity along the limb. Mystery remains!
Our observation of unit intensity ratio certainly is consistent with Minnaerts reference to Schoenberg's statement - namely that the 'intensity along much of the bright limb is a constant value'. Intensity ratios between limb points would give unity.
From flux to AlbedoPosted by Daddy-o Sep 04, 2013 01:48PM
In 1941 Minnaert published a paper
about lunar photometry - in particular the reflectance. There is an idea in that work which we can use to view our data, and perhaps learn something.
The intensity we observe in any pixel of the lunar disk is:
Intensity ~ albedo * reflectance(i,e,alfa)
Where 'albedo' is a measure of the darkness of the material in the pixel and 'i,e,alfa' are angle of incidence, angle of emission and lunar phase, respectively.
Consider now two points A and B on the lunar disk placed such that iA=iB and eA=eB. Furthermore we have that the phase, alfa, is almost constant all over the image (alfa is the angle between a point on the Moon and the observer and the Sun - and apart from the finite size of the Moon compared to the distances to Sun and Earth this angle is the same for all points on the image of the Moon). In these two special points we thus have
reflectance(iA,eA,alfa) = reflectance (iB,eB,alfa) so that the ratio of observed intensities from A and B are simply:
I_A/I_B = albedo_A/albedo_B
By finding many pairs like A and B we could investigate how ratios of albedos across the Moon vary - or compare the ratio from observations to our map of albedo - at the present "Clementine scaled to Wildey".
We now do just this!
From the synthetic image software code that Hans wrote we have, for every pixel on the model image, its angle of incidence and emission. We can therefore select an arbitrary point A on the BS and find its symmetric point 'B' also on the BS, extract the observed intensity ratio A/B as well as the same ratio for the ideal model image and tabulate these against solar zenith angle (i=SZA) and earth zenith angle (e=EZA). We have done this for 1000 randomly distributed points and show the results here:Top panel shows observed (black) and model (red) A/B ratios plotted against EZA. Small EZA (in radians) correspond to points near the middle of the terminator on the disk while large values correspond to points closer to the limb, near the sky. Second panel shows the same plot but now using solar zenith angle (also in radians) as the x-axis - small SZA correspond to points near the sub-solar point, in this image near the limb, while large SZA correspond to points near the terminator where the Sun is low in the sky as seen from the Moon. Bottom left plot is the ideal ratio against observed ratio. Green line is the robust regression line and blue line is the diagonal. Bottom right is a small image of the Moon with points A and B plotted as white dots.
We emmidiately notice the tendency for observed ratios to be larger than ideal ratios - at least for the median to large values of the ratio: the observed ratios lie above the diagonal line in bottom left panel.
The observed ratio has a larger span not just because of 'noise' - careful inspection of 'slices' across e.g. dark Mare [not shown here] reveals that observed A/B ratios reach more widely separated extremes there than does the ideal ratio slice across the same slice.
This seems to imply that the albedo map we use is too 'flat' - it should be scaled so as to give more extreme darks and lights by perhaps 25%, judging from the plots.
We also have to wonder why albedo ratio is a function of EZA and SZA - nearer the terminator or middle of the disc ratios are simply higher than near the limb. As intensities are smaller there while brighter near the limb we wonder what is going on - remember that the ratio is reflectance-independent by construction! Note also that both observations and models behave in this way. It cannot be 'nonlinearities in the camera' since the model behaves in the same way, nor 'scattered light in the observations' since the scattering-free model behaves in the same way. The model is also assuming a spherical surface - no hills, boulders or crevices for shadows to hide in.
Here is a repeat of the above for a different lunar phase:
Similar features are seen here - towards small SZAs the ratios go to 1. As before the general picture is that variations observed from dark to light are larger than in the model. This reminds us that we are using a fixed albedo map in the synthetic model - we can expect different dark/light ratios in different wavelength bands. The above scalingmay sereve to achieve this?
We shoudl repeat for many examples from each band and each lunar phase to see which features are general.
There is a comment by Minnaert in, "Planets and Satellites" ed. Kuiper and MIddlerhurst, Vol III. p. 222 to the effect that the "bright limb shows the same brightness over 3/4 of its length and that this was noted by Schoenberg (1925) and this was only rarely checked" - wonder if that last remark is still true? We can certainly confirm that the albedo-ratios tend towards 1 near the limb, but we do not understand yet if these two observations are linked. Time to read Schoenberg!
Post-Obs scattered-light rem.Posted by Daddy-o Aug 30, 2013 03:06PM
Our observed images are a convolution of a point-spread function (PSF) and the Object (the Moon). We know from analysis of images of various sources - the Moon itself, Jupiter, and stars, that the PSF has the character of a power law along the lines of a core with wings that follow 1/r^alfa with alfa near 2.9.
It is in principle also possible to determine the PSF by de-convolution. Clasically we would expect that
Image = Object * PSF
Where the '*' implies spatial convolution. Taking FT of both sides we get
F(I) = F(O) x F(P)
where F is the forward Fourier transform, and the 'x' implies multiplication. Rearranging and taking inverse Fourier transforms (f) we get:
P = f(F(P)) = f(F(I)/F(O))
We have I (the image) and if we had O we could calculate the PSF P. Usually P is very noisy because of amplification of small noisy signals at high frequencies due to the division above. We can average over several estimates of P, however.
For us, this is easy because our data come in the form of 100-image stacks. Unfortunately we do not know O, the object. But we can make synthetic images of O that are highly realistic!
We have applied the above procedure on an image-stack of the Moon, for which we also have a synthetic image. We average the 100 PSFs that are generated. The radial profile from the peak looks like this (black curve):
The blue line is a power law with alfa=2.1 and the red line is a power law with slope 0.3. For this particular night we know that a single power law (plus a narrow table-lookup core) of alfa=2.6 has been estimated for this image using forward modelling techniques where we fit parts of the scattered halo at about 100-150 pixels distance from the photo-centre of the bright side of the Moon.
The two estimates of the PSF are thus quite different, and I wonder why.
Possibly it is a sign that the edge-fitting can be performed well with a large family of PSFs. We know that the fit is excellent at the edge of the disc.
I suggest that a next step could be to test both PSFs or types of PSFs and see how they perform in various situations. How close to the observed image is the ideal image once it is convolved with the above profile?
Exploring the PSFPosted by Daddy-o Aug 28, 2013 02:42PM
We have indications that the halo seen in B and V images are somewhat similar, but that VE2 images of the Moon have decidedly different halo profiles. Because the filters used in B, V and VE2 are fabricated (and function) differently we may have the situation that this causes the differences in halo profiles. B and V are 'coloured glass' filters while VE2 is a thin-film filter. In a DSLR detector the R,G and B filters are essentially bits of coloured semiconductor material. As a test we inspect the halo seen around the Moon in images taken with a DSLR camera. In such images the R,G and B channels are obtained at the same time at precisely the same observing conditions.
On the internet (http://bit.ly/13YZIOU
) we have found a very detailed large-scale JPG (i.e. 8 bit only) 10-min exposure image of the sky near Orion containing the Moon. We submitted the image to nova.astrometry.net and received a solution back, including image scale. We took the resulting WCS-equipped image and extracted the profile of the lunar halo in R, G and B and plot these against radius from the estimated disk centre. We have subtracted a sky level for each colour, estimated by eye, and show the profile in the upper panel and repeat it in the lower panel with straight lines fitted:
The profiles are saturated out to about half a degree but after that they follow a remarkably similar shape in this log-log plot. Fitting power law functions (1/r^alfa), we may even see some sort of 'straight line behaviour' between radius 0.6 and just short of 2 degrees, with another linear trend taking over out until the sky-noise is reached at 4 degrees or so. The slopes are fitted-by-eye only but are -2.9 (near the canonical 'can-never-be-steeper-than-3') value, and -1.3.
Before assuming that 'DSLR RGB imaging' will solve all our filter problems, let us recall that the VE2 vs other profile differences we have found are subtle; that the above is based on an 8-bit image; that nothing is known about the image treatment performed by the author of that image, and that we do not have data similar to VE2 here - the R channel is not VE2 [ ... but read more here link
!]. Let us instead try to do something similar with 14-bit RAW images.
Note that the above image was obtained by tracking the sky. Apparently, tracking allowed a long exposure that gave us a wide halo to study. In our own DSLR-On-the-sky wide-field images we have failed to get results similar to the above - exposures were limited to several seconds to avoid overexposure and trailing. Our own MLO telescope images are restricted to the 1/2 degree reach from disc centre.
Note that the above essentially speaks very well for the sort of DSLR optics used - the amount of scattered light near sources must be low
or we would not see so steep a PSF! Again, this may be easier to do with a wide-field lens such as used for the above image - trhings may be different with a tele-lens that allowed a closeup of the Moon. The 'core' of the PSF should be better investigated, if we can get some of our own images of e.g. Jupiter or bright stars - the above image is VERY wide-field and contains zillions of crowded stars, not likely to give us good point-source PSFs.
Met sensorPosted by Daddy-o Aug 08, 2013 09:36AM
We have mounted the Aurora weather sensor on the roof of DMI and found out how to run it from Linux. A plot is generated automatically and updated each minute. The link is here:http://web.dmi.dk/solar-terrestrial/staff/thejll/metdata.html
Top frame shows sensor temperature (i.e. how hot the sensor itself is) and the radiance temperature of the sky measured inside a wide cone looking up.
Next frame shows the difference between the two - when it is large the sky is probably clear - i.e. no clouds and no humidity.
Third frame shows the amount of light reaching the sensor - so you see day and night and clouds passing in front of the Sun.
Bottom panel shows rain falling on the sensor.
Future work will now be directed towards producing and statistically validating a warning signal ('Close The Dome!') on the basis of these readings.
We might mount a webcam also.
Showcase images and animationsPosted by Daddy-o Aug 07, 2013 01:38PM
We have the unique B and V images from JD2455945.17xxx in which the 'halo' cancels almost perfectly, allowing us to see the colour of the DS. At the time of observation we can ask what the Earth was like. Here is the image generated from the Earth Viewer at Forumilab ( http://www.fourmilab.ch/cgi-bin/Earth ):Above
is the model image of Earth seeen from the Moon at the time /Jan 18 2012, about 1600 UTC) of our observations. Below
is imagery from the GOES West satellite. There are also GOES images from 12 UTC and 18 UTC, the above one is from 15UTC and is the closest in time.
Evidently the Moon and the GOES satellite were not in line as the aspect is slightly different but we get the idea: Southern Ocean and Antarctica was white with cloud and ice, North and South America were partially cloudy, and the sunglint was on ocean off the coast of Peru, approximately.
Here is a simple model image,
as if seen from the Moon at the relevant time. The image is generated from geometry and the 'NCEP reanalysis cloud product'. You notice some similaririties in cloud cover between the satellite image above and this model image. The model only does the perspective projection of pixels onto a sphere - there is no modelling of limb brightening or anything like that.
The images does allow a simple analysis of the effects of various extreme values for land albedo, and used for two assumed wavelength bands could be used to set limits on what we expect the colour of the earthshine to be at this particular time. More to follow!
Observing logPosted by Daddy-o Aug 02, 2013 12:29PM
Ingemar was here and looked at the LV code. We semeed to learn the following:
1) The NI Report Generation Toolkit license has run out and needs replacement.
2) The newest set of LV codes are on the NAS under 'Earthshine Project/LabView Sou.../'
3) The Engineering mode refuses to run because of some error messages related to the AXIS - these are for the FWs, the SKE and the Dome. No matter what you want to do with the Engineering Mode it will always start by initializing all its sensors, and if one of them fails, nothing
works! Since we have no dome there will indeed be a problem here!
Ingemar was able to turn off some of this (NB: set it back to as-was, later!) and indeed the code progressed to a further point (when it then ran into other problems). Front shutter was tested while the error-checking was turned off in Eng. Mod. but did not work.
Of the above 1) may not be important as long as scripst are not being read. But 3) seems pretty serious.
Are the new licenses for the MAX correctly installed?
Can the code that refers to the dome be turned off?
Will things work again once the dome is turned off and the other AXIS cables are plugged in?
Why did front shutter also not work - it has no motor?
Some of the above is now clearer. See here
. A number of the errors were related to wrong LV license number, and the new IP numbers for DMI must be set insid ethe LV code. For reasons we do not understand the COM-port numbers had changed.
Post-Obs scattered-light rem.Posted by Daddy-o Jul 11, 2013 03:47PM
Measuring B-V on the DS in B and V images with different PSFs: The Case For Forcing them!
We have detected some image-pairs where the B and V halo appear to cancel. We have plenty of other image-pairs where they evidently do not cancel. This is seen by structure on the sky - DS sky in particular as well as 'slopes' across the DS in the direction towards the terminator near the BS. We here investigate whether it is possible to 'force' the image with a narrower PSF to acquire a halo that resembles the broader image's halo so that cancellation can occur.
If the differences between the PSF width-parameters (alfa) is small we expect to convolve the sharper image with a quite narrow PSF that only broadens the halo a little. We will use PSFs that are based on our 'canonical PSF' and simply raise it to a large number - this will generate a more and more delta-function like peak.
We identify centered pairs of B and V images that are close in time; identify the image with the narrower PSF - by using previously fitted alfa values; we then broaden the sharper image, while conserving flux, and inspect the difference between the B and V images. The results of such a trial is shown in the Figure:
Here is a less fuzzy pdf file of the above plot:
We see 15 panels showing a 'slice' through the disc centre, of B-V (magnitudes). Each panel has an increasing power (alfa) used to raise the canonical PSF to - at first we raise it to 1.5, then to 1.6, and so on. A smaller power will broaden the PSF a lot and thus give us 'fuzzier' images. We expect that when the image is too fuzzy we will see slopes in the B-V slice - just as when two observed images in a pair are subtracted and one has a broader PSF than the other. As alfa increases in the figure we reach a point where it is so large that the sharper image no longer is made appreciable more fuzzy - in the limit of large alfa it becomes an almost identical copy of the original image - hence, towards the end of the sequence of plots we essentially see what the slice across the original B-V image looked like.
In each image we notice the DS to the left and the BS to the right - both situated between the two vertical dashed lines, denoting edge of disc. We see a 'level difference' between the DS and the BS. We notice the slope of the DS - upwards for large alfa and downwards for smaller alfa - e.g. at alfa=1.7. Somewhere near alfa=1.8 and before alfa=1.9 the slope is horizontal.
We expect the real DS to be 'flat' [we need to check this against realistic models] so when alfa is near 1.85 it seems we have induced enough additional fuzziness in the sharper of the two images that the halos cancel and the level difference between DS and BS is what it is in reality, [We need to test this on images with known properties: Student Project!].
We notice also that the level difference depends on alfa, even when alfa has gone past the 'now DS is flat' point. This is a warning that incorrect estimation of
the limiting alfa value may give us spurious results for delta(B-V)! We estimate the slope on the DS (between vertical dotted lines) and look for the values of alfa where the DS slope is not statistically different from zero at the 3 sigma limit. We also do this for the BS. We find midpoints of the fitted DS line and the fitted BS line (red lines in uppermost plot) and take the difference bétween these, forming a sort of '+/- 3 sigma interval of confidence' for the B-V difference between BS and DS. This is plotted here:
Here is the pdf:
We see thaat the generous +/-3 sigma interval allows quite a range in B-V - and also that if we do not estimate the right alfa in the above procedure - but make it too large then we will underestimate B-V.
For the 3-sigma limits we squint at the above graph and read off upper and lower B-V limits for the BS/DS difference: 0.5 to 0.7.
All of the above has been done for an average of an 11-row slice across the middle of the disc. The method could in general be extended to treat broader strips, or the whole ds - planes could be fitted instead of lines.
First, though, it is necessary to understand if the above is even correct! We need to look at artificial images of the Moon. These should have their own "B-V colours" then we fold the B and V images with seperate PSFs and try to force one to be as fuzzy as the other, and so on. The actual slope of the DS, when fitted with lines or planes could then be estimated and in fact used as the goal slopes for the above procedure. Work work work!
Real World ProblemsPosted by Daddy-o Jul 05, 2013 03:44PM
We have noticed that some images have a JD time that differs from the JD implied by the time information in the FITS file header. By checking (almost) all files that have a JD in the filename against the time in the header the following list of suspect images are revealed:
This list is the JD from the filename - the value is 1 hour larger than the JD implied by the DATE field in the FITS header. None of these appear on the list of good images as determined by Chris. and his 'tunelling' code which inspects magnitudes vs lunar phase.
Met sensorPosted by Daddy-o Jul 04, 2013 09:06AM
On JD2455945 conditions were such that the BS halo in the B and V images cancelled almost perfectly, giving us the possibility of seeing the DS colour itself. We have a link to material discussing scintillations here
Is it possible to understand which meteorological conditions led to this unusual situation? At the MLO there is a meteorology tower and data are taken and stored for every minute. A link to the data is here
A plot of selected parametrs for the night in question is here:
The plot shows 3 days on either side of the observation moment. A more legible pdf version is here
a) The rising pressure. The daily cycle is due to heating of the atmosphere,
b) that the observation occured towards the end of the night - temperature at ground level was dropping,
c) the vertical temperature gradient was positive (it was indeed almost the maximum seen that night) - the air was warmer higher up,
d) wind speed was relatively low,
e) relative humidity was relatively low.
We next plot 'alfa' (the parameter that sets the width of the PSF in our model images, found by fitting), against four of the above parameters:
There seems to be no strong pattern. Slightly, there may be an indication that low vertical T gradients allow for larger alfas, and that low relative_humidity allows for larger alfas. Ignoring the outliers at low alfa (a sign of a very poor fit or night) we look closely at the dependence on relative_humidity:
It would seem that for RH between 10 and 20% a large value of alfa is obtainable. Large lafa implies a narrow PSF. However, fo rthe nighjt JD2455945 (where the halos cancelled) we have unremarkable conditions: alfa is small (1.54), pressure is medium (680 hPa), relative humidity is medium (42%) - only wind speed is lower (3 m/s) than most other data points. Th ethree values of alfa always determined from the same image are very stable, however, differieng by 0.001 only.
Speculating wildly: Is it because alfa is small that we get halo cancellation? With a small alfa the halos widen and perhaps their 'tails' cancel better?
Here are some papers that discuss meteorological conditions and 'good seeing' conditions:
Post-Obs scattered-light rem.Posted by Daddy-o Jul 03, 2013 10:29AM
Is it possible to ab initio
calculate the expected colour of earthshine? Earthshine, in the visual band, is due to diffuse scattering from clouds and the surface of the Earth, and Rayleigh scattering from molecules in the atmosphere. Can we simply calculate the expected B-V colour of Earth, seen from space?
If we assume that the diffuse scattering on clouds preserves the colour of the light (I think this is true - clouds look 'white') and if we assume that clouds cover most of Earth (a fair assumption given that the oceans are dark and cover 70% of the Earth) then we can make a model of Earth's spectrum consisting of a fraction of the solar spectrum plus the complementary fraction of light 'blued' by Rayleigh scattering. This model assumes no 'true absorption' in the visible bands (also a fair assumption; ozone may actually absorb at UV wavelengths but the solar flux at those wavelengths is small).
By considering the Rayleigh effect and imposing flux conservation we get the following B-V colours for the spectrum of Earth:
where 'f' is the fraction of the beam that undergoes Rayleigh scattering. We have elsewhere estimated the B-V colour of Earth (here we do not mean 'earthshine as measured on the DS') to be 0.49 +/- 0.02. That would seem to imply that we have a correspondence with (crude) theory if 10% of the beam undergoes Rayleigh scattering.
Since most photometrically measured extinction coefficients are in the 0.1 - 0.2 mags/airmass range it does not seem unlikely.
In meteorological observations pyranometers
are sometimes used to monitor the clarity of the air during the day - they typically measure the global radiation (all wavelengths, all directions - i.e. from the Sun and from the sky) and the 'diffuse' radiation - i.e. only the part of the light not coming directly from the Sun. Perhaps it is possible to estimate the fraction of light removed from the incoming beam, by scattering, using such, essentially bolometric measurements?
Using a SURFRAD
pyranometer data set from Boulder CO, USA, I determine the noontime ratio of Diffuse to Global radiation, and plot it for each day of a whole year:
There is a great deal of variability! This is due to clouds. On perfectly clear days I guess that the ratio of diffuse to global radiation is at a minimum since the contribution to the diffuse is only from the blue sky while the decrease of the direct is also only the Rayleigh. Clouds tend to increase the former and decrease the latter giving a larger ratio.
The incoming beam contributes to the diffuse radiation and to the global radiation. The same amount of light scatters down to Earth as scatters into space so:
Incoming = Global + Diffuse
Then D/I = D/(D+G)
Looking at the plot it is clear that near the middle of the year you
can see D/(D+G) near 0.1. This is not exactly the same as saying 'the
bolometric extinction coefficient is 0.1' but we are close.
Problematic is the large amount of clouds - I shall look for a station that has less clouds!
It seems that Desert Rock Arizona is less cloudy:
Notice that we are getting D/(D+G) as low as 0.07. The altitude of Desert Rock AZ is 1007 m, while the station near Boulder, above is at 1/4 the altitude (213 m). Does that explain the different minimum ratio?
At MLO they publish a transmission coefficient which is approximately the same we are trying to look at - it would seem that at MLO the transmission can be 92-94% when there is no volcanic dust: http://www.esrl.noaa.gov/gmd/grad/mloapt.html
Bias and Flat fieldsPosted by Daddy-o Jun 17, 2013 09:03AM
We rely on subtracting 'smooth scaled superbias' fields from science frames, because of the 20-minute thermostat-oscillation in the bias mean, and a wish to avoid adding noise by subtraction of noisy bias fields. We generate the scaling factor for the superbias by averaging the bias frames taken just before and after the science frame. We seem to be assuming that the bias mean does not change - or does not change irregularly - during this procedure. Let us examine that assumption.
We take most bias frames as single frames, but happen to have 50-image stacks here:
We calculate the average of each subimage in these stacks and plot the results:
The camera can take many images per second in 'kinetic mode' so the above sequence last something like 10 seconds each. There is evidently some slight trend during that time - at the 0.05% (or 0.15 counts) level compared to the bias mean. This may not seem much, but if the scaled superbias mean is wrong by 0.15 cts it can mean an error on the DS flux of many percent - because the DS fluxes we are using are at anything from near 1 count above sky to maybe 10 counts above sky. For a DS at 1 count the 0.15 count error is 15% while it is 1.5% for the 10-count DS. This is quite serious.
It may be the reason that the halo-removing methods based on 'subtracting a model halo' did not work well - they relied on the bias being correctly subtracted previously. The 'profile fitting method' apparently worked better and this is probably because it inherently contains a fit to the sky (and thus bias, and also any insufficiently subtracted, bias).
Bias subtraction does more than subtract a mean level, of course - there is also some structure - notably a 0.2 count raised edge along the vertical sides, stretching some 10% into the field. Some row asymmetry is also evident, along with a very slight 'ripple' in the column direction (Henriette's thesis has all this).
Subtracting the scaled superbias is therefore mostly a good thing, but it should be realised that important level-errors may be present causing blind reliance on the 'complete bias removal' to be dangerous - better to allow for an additional small pedestal in your modelling.
Post-Obs scattered-light rem.Posted by Daddy-o Jun 14, 2013 10:02AM
In considering B-V images, as here
we are have to know how far the BS halo reaches onto the DS. We have looked at that before, here
. We now revisit this issue. Here is a contour plot of the DS and the halo, with colours so that we can see how far the halo is likely to reach - we see it reach into the sky on the BS - how far do we think it reaches onto the DS?
Consider the yellow end of the greens, for instance - on the sky that contour lies 40-50 pixels from the BS rim - on the DS it lies adjacent to the red area which is the BS. The yellow-green contour therefore does not interfere much with most of the DS. The blue contours on the DS, howvere, are represented on the sky far away from the BS - so the blue areas on the DS may be interfered with by that part of the halo.
The above is V-band image. In B-V images we rely on much of the halo being similar in B and V and thus cancelling. The above noticeable gradient in DS brightness is absent in the B-V image.
Bias and Flat fieldsPosted by Daddy-o Jun 10, 2013 01:01PM
These are calibrated B-V images from JD2455945.17, as explained at this link
The one on the left is generated WITH flatfielding of the B and V images used, while the one on the right is generated WITHOUT flatfielding. They are almost identical, but not quite. Here is the difference between the two:
The pattern is the expected pattern from the CCD chip. The histogram of the values is here:
So there is a difference which is distributed around 0 with S.D. 5 millimagnitudes. The effect of flattening would appear to be small - but recall that these are difference images.
Apart from various problems related to aligning images it is rather nice to work with difference images - problems tend to cancel out! However, note in the above how some of the structure on the DS in the B-V image looks like the features of the falt field - as if perhaps the flatfield we used did nothave enoughj 'amplitude' in the stripes.
Post-Obs scattered-light rem.Posted by Daddy-o Jun 06, 2013 02:20PM[Update: See note at end!]
post Chris showed that the night of 2455945.1xxx has halos, in B and V, that seem to be at a constant distance across the DS. This implies that the 'gradient problem' is absent on this night - perhaps because the night was very clear or the haloes in B and V almost identical. [Inspection of the 'alfa' for the profiles shows that V images had alfa in the range 1.547--1.549 while B images had alfa in the range 1.544--1.547. These values seem close but halo shape is very sensitive to alfa, so analyze first!]. It is possible to generate a B-V image for this night using the only good images for B and V (in the sense of 'being on Chris list of good images', as explained elsewhere). We used one B and one V image:
both bias subtracted but not flat-fielded. Exposure times were taken from the image headers.
This is the result:
The colours are of course chosen arbitrarily, to aid excitement and imagination! For this image we corrected for extinction using
Vinst(idx) = -2.5*alog10(Vim(idx)) - Vam*0.1 ; kV=0.1
Binst(idx) = -2.5*alog10(Bim(idx)) - Bam*0.15 ; kB=0.15
and calculated B and V images from the B and V instrumental images, in an iterative manner:
BminusV=Vinst*0.0+0.92 ; BminusV is an IMAGE
for iter=0,10,1 do begin
V = Vinst + 15.07 - 0.05*(BminusV)
B = Binst + 14.75 + 0.21*(BminusV)
Here, 'idx' is a pointer that selects for all pixels on the lunar disc - both DS and BS.
Vonvergence was swift and independent of initial value. Convergence was to B-V=0.925.
A plot of a 'slice'a cross the disc above is here:
We see a BS B-V value near the dashed line at 0.92 and a DS value between 0.75 to 0.8.
The colour image reveals some structure on the DS. It seems we see a faint flatf ield pattern? This should be looked at. On the BS colour differences eem to be lunar in origin and coul dbe compared to the literature on this subject.
The deep 'cut' in the middle is due to some small-distance difference in the B and V halos, we think.
Until we are told otherwise we think this is the first map or image showing the colour of the Dark Side of the Moon, which is also an album by Pink Floyd
Images in other colours may be fothcoming.
Another version of the image above, here with a legend is here:
On the night of JD2455945.1ish the Earth was iilluminated over South America and shone on the Moon. Seen from the Moon the Earth looked like this, from the Fourmilab Earth Viewer:
So, mainly Pacific Ocean but also also all of South America. Need some satellite images of this to see how the clouds were distributed!Note added later: The (far) above is unfortunately quite sensitive to how well the images are aligned. This needs to be looked at before anything else.Here is a nice link to images of the
colour of the Moon obtained in sunshine and with simple DSLR cameras -
i.e. jpeg files: http://www.datarescue.com/life/kepler/moontests/raw_vs_jpeg.html
Observing logPosted by Daddy-o Jun 06, 2013 10:23AM
Here is a list of non-lunar images taken at MLO. There are stars, clusters, galaxies, and planets and asteroids. Many are suitable for studies of extinction.
Showcase images and animationsPosted by Daddy-o May 23, 2013 10:47AM
Here are some images of the telescope assembled - perhaps it is possible to use the images to understand which plug goes where? Sharper jpegs are in /home/pth/SCIENCEPROJECTS/EARTHSHINE/JPEGS/ - they are the 'imgXXX.jpg' files.
I've also added images from the construction in Lund - those images are in same place as above, but are called imagexxx.jpg (xxx from 332 to 347).
shows more images from the telescope on MLO.
Here is an image of the cabling at the back of the CCD, during MLO.
Real World ProblemsPosted by Daddy-o May 23, 2013 09:30AMLargest box:
Telescope tube plus its attached cables.
Rack of electronics.
Very long and thick bundle of cables.
Rod for Hohlraum source + footSmaller Wood box:
LCD monitor + VGA cable
Laser printer Samsung ML-2525
Small white box: Spare lamps + relay
Holhraum Sphere in its own box
Box of fan filters for rack
2 x Shutter boards + Edmund pack of flat soft things + a small mw laser + a filter labelled 'IR cut'
European power supply for Axis camera
2 Axis cameras with robofucus lenses, and additional lenses and mounting brackets.
Rain and light sensor - DMI?
Väisala device in 'chineese tower' affair.
DMI IR rain sensor
Väisala humidity sensor HMT100
Cables + spare PXI parts + bracket for adjusting SKE
Uniblitz shutter # VS25ST1-100
Vincent Associates 710P Shutter Interconnect Cable
3 Thor labs FW drivers / electronics. Model FW102B. Implies that we have
three Thor labs FWs - the colour FW, the ND FW, and one more - somewhere!
Box of mixed cables - VGA + DVI + 2xUSB repeaters
2 sealed metal envelopes with "D.C.D. Panel Mount SA 4 COMP"
Tools for shutter tuning
Spare Ferrorperm Knife edges on glass.
Mixed VGA, DVI, USB cables along with 2 USB extender cables
US 115V power cables - to PC-type plugs.Blue box:
Polar alignment scope
Point source lamp + scope
Magma + mounting HW
Dome breakout box
base w. electronics
Power for Axis camera (perhaps it moved to Wooden box?9
A part of the PXI - probably the spare or original PC
Spare HD for PXIWhere are:
Mount handbox controller?
Aurora cloud sensor?
Various interface cards for camera to PXI?Additional
Real World ProblemsPosted by Daddy-o May 23, 2013 08:34AM
Here is a collection of links useful in our attempts to power up the rack of control system, at DMI:
iBootBar : http://dataprobe.com/support/index.html
Optical designPosted by Daddy-o May 21, 2013 07:58AM
Here are example sof a 'ghost' and 'dragging':
The dragging is due to no shutter being used - i.e. readout was only way to terminate exposure and hence frame was illuminated as the image was shifted to the 'hidden register' We hav a frame-transfer camera. The image is from the test phase in Lund and no shutter was installed then.
The BS ghost is faintly vissible: It is a copy of the BS and the lower cusp is seen poking out slightly down and to the left of the real BS. This was due to the CCD camera being aligned almost perfectly along the optical axis of the system - the ghost arises as the shiny CCD surface reflects light back into the optical system and reflections are produced at all optical surfaces - the back of the second secondary lens, the front of the second secondary lens, the back of the first secondary lens, the front of the first secondary lens and the back of the primary objective and its various surfaces. At MLO the camera was tilted at an angle so that the reflection from the CCD did not go back up the system. This tilting did not appear to have any adverse effects on focus etc.
Images from lab hard disc recovered from MLO.
From flux to AlbedoPosted by Daddy-o May 06, 2013 03:42PM
Some of the notes discussed below, in the next several postings, are collected in this doc:
There are some considerations of how how much albedo change we expect during global warming - and the detectability of such changes are discussed.
From flux to AlbedoPosted by Daddy-o May 06, 2013 09:17AM
But we - and BBSO - calculate albedo by comparing the earthshine measured on the Moon to the intensity of earthshine predicted by a terrestrial model based on a uniform Lambert sphere.
We test how well this works by taking a series of GERB satellite data for several weeks across a year and extract the total flux from the whole-disk images. The MSG satellite bearing the GERB instrument floats over lon,lat=0,0 so always sees the same part of Earth. Johanne has extracted images for every fifteen minutes for several weeks in a year. We plot that (top panel, below).
We use the eshine synthetic code Hans wrote to generate Lambertian images for a full cycle of sunrise over earth. As the code is based on what the Earth looks like from the Moon we pretend that one month is like one day and thereby can extract the phase law for the Lambertian uniform-albedo Earth in order to compare it to the GERB data. We also plot that (second panel, below). [Note that the difference between view from satellite and view from Moon may be important: Sun-Earth-Viewpoint angles should be the same, and if the Sun and Moon are not at similar latitudes as Sun and Satellite we could be generating artefacts in what follows: we should see if we can use the synthetic code for satellite viewpoints. For now we ought to find dates when the Moon was at latitude 0 (like the satellite) and the Sun also same latitudes - tricky to do.
Lastly we divide Gerb fluxes by Lambertian fluxes, correct for the fact that Geostationary orbit and the Moon are at different distances and multiply that corrected ratio by the uniform albedo used in the Lambertian models. We plot that (bottom panel, below).
We see that the albedo does not come out constant. This is not surprising since the Earth has real clouds that drift around - but that is only what gives the thickness of the thick line of points in the last plot above. The 'wiggles' are due to the inadequacy of the Lambertian model. Near New Earth (Full Moon: never observed) the derived albedo rises. Near Full Earth (New Moon: attractive to observe due to strong eshine, but difficult due to Moon close to Sun) the albedo is flat. At intermediary values (20% and 80% of the cycle) the Lambertian albedo is relatively high so that the derived albedo is lowered.
How can we use these insights to understand what Johanne shows in plots of how derived albedo evolves during the nights?
Our aspiration is that the above can give insight into
1) the 'phase dependency' we see in derived albedos when we plot all data corresponding to all phases during the morning branch - i.e. Moon setting over Western Pacific/Australia, and
2) the nightly tendency to have falling albedo through the night, for that same branch.
As for 1 the reader should look in the May 3 presentation at slide 22; as for 2 the reader shoul dlook at slide 23.
We have to figure out whther the above plots explain any of these sightings. Could the almost quadratic phase dependency seen when all data are plotted be due to the 'dip' near 20 and 80% of the cycle? Could the 'nightly slopes' be due to the same?
Showcase images and animationsPosted by Daddy-o May 03, 2013 12:31PM
This earthshine presentation was given at DMI, on May 3 2013.
Post-Obs scattered-light rem.Posted by Daddy-o Apr 29, 2013 10:29AM
We have previously
considered B-V images of the Moon. This was done with 'raw' images - that is, images where the halo had not been removed. Since we have the BBSO linear method implemented and since it does clean up the DS we can also calculate B-V images for the Moon based on these.
We have 55 pairs of B and V images thata re close in time (about 1 houror less apart). Using the standard star calibration relationships that Chris worked out fropm NGC6633 standard stars, we can convert images to instrumental magnitud eimages and from there to calibrated B and V magnitud eimages. We also corrected for extinction since the images were not obtained at the same time.
SInce the calibration relationships depend on B-V we have to assume some B-V values and iterate (Chris solves algebraically). The iterations converge quickly. We use onlythe brightest pixels in each image - i.e. the pixels delineating the BS - for calculating the mean B and mean V values needed to update B-V in each iteration.
The values for BS B-V that we converge to have this distribution:
The mean B-V=0.989, and the S.D.=0.019. The accepted value - e.g. Allen (4th ed), Table 12.14, gives B-V=0.92 (van den Bergh observations?). We therefore have a significant discrepancy. It should probably be noted that our values come from phases near 90 degrees, while the Allen values may be from 'Full Moon' conditions.
If we accept the above B-V (BS) values at face value we can continue:
The Sun has B-V=0.642 (Holmberg et al, MNRAS 2006). One reflection off the Moon reddends this value by 0.989 - 0.642 = 0.347. This value will also apply to earthshine that is observed after one reflection off the Moon, even if it is the DS. [We ignore here any colour-dependencies in the lunar surface mare vs highlands!
If we can estimate the B-V of earthshine as seen on the lunar surface, we can work backwards to what the B-V of that light was before it struck the Moon - it will be the observed value minus 0.347.
Before trying this we need to understand to which degree the use of BBSO linear images, as opposed to 'raw' images, has helped us observe the true colour of the DS - has an important amount of the BS halo been removed from the DS?
We generate centered B-V images and plot the average of 20 rows across the middle of the images:We see two panels - each panel is the result of using a fixed B image and two different V images - all three taken a short time apart. The black curve is the run of B-V values in the 'raw' image - that is, the image where no effort has been made to remove the BS halo. The red curve is from images cleaned with the BBSO linear method. The deep jag in the middle is the BS/DS terminator. The DS is to the left of this and the BS to the right. Since the BS is not altered by the BBSO linear method the red curve covers the black curve on the BS.
On the DS we see that cleaning the image has resulted in a slight reddening of the DS - it was 'too blue' in the red images.
We also see that the 'linear gradient' in B-V across the DS is unaltered qualitatively by the cleaning of the image. Why?
If we push on, ignoring the not-yet-understood gardient, and assume that the part of the DS closest to the sky has an un-polluted B-V value then we can calculate the colour of earthshine before it strikes the Moon, as explained above. First we extracted DS B-V values for that part of the DS disk that is to the left of 90% of the vertical columns on the disk. These values were on average 0.29 +/- 0.05 below the BS value.
If the BS value is given the canonical B-V=0.92, then we have a B-V for the DS of 0.63.
Franklin (JGR 72, no 11, p.2963-, 1967) measured B and V repeatedly on the DS. The difference between his mean B and his mean V is 0.64. We are close, but we are worried about scattered light!
Subtracting the effect of reflection once on the Moon brings us to the value for B-V of earthshine, before it strikes the Moon, that is, as it would be seen in space:
B-V_ES = 0.28.
There is one published B-V value for earthshine, based on Mariner II data in the 1960s. The paper is http://adsabs.harvard.edu/abs/1964JGR....69.4661W by Wildey. Unfortunately I cannot make head or tail of that paper!
Playing a bit more with the above, we can consider the effect of Earth on light - the Sun has B-V=0.642 when it strikes Earth. If the earthshine has B-V=0.28, then the bluing effect of Earth is 0.28-0.642 = -0.36 in B-V.
Real World ProblemsPosted by Daddy-o Apr 24, 2013 10:34AM
We update yet again the list of 'best images' that Chris has generated by inspection of compliance of absolute magnitudes against lunar phase.
We can removea few more images by hand inspection. We found about 10 that have 'cable in view' as well as various near-horizon problems. The list is here and now contains 525 images:
We note that Johanne is working her way through many images and finding 'bad focus' cases. Since we believe these are coincident with 'not the right filter acquired' cases, we shall eventually be further updating the list of best images.
Student projectsPosted by Daddy-o Mar 22, 2013 09:51AM
We have often come across good ideas for student projects. Here is a start of a collection of projects - just links, but text can be added to explain more.
How do meteorological conditions determine seeing at the telescope?
Was the bias pattern
Understanding the PSF:
Albedo maps and their use in modelling observations:
Atmospheric turbulence studied via Moon images:
Colour of earthshine - Danjons work:
Image analysis methods - Laplacian method:
Exploring the PSFPosted by Daddy-o Mar 22, 2013 09:30AM
post we saw that the difference between B and V (magnitude) images could have the shape of a linear slope on the DS and plateau on the BS. We are trying to recreate that using synthetic models. It is surprisingly difficult!
Using V and V images we saw that differences typically had the shape of level offsets - not slopes. In the B-V images we saw linear slopes on the BS. I thought the linear slopes originated in different PSFs in two filters - different alfa-parameters, for instance.
Well, taking a synthetic image and convolving it twice with two slightly different PSFs and converting to magnitudes and subtracting gives this:
Upper panel shows the ideal image we are using - BS to the right and the rest is DS. Bottom panel shows the difference between the image convolved with alfa=1.73 and alfa=1.72*1.02. DS is columns left of 360 - there is no linear slope. There are plenty of features on the DS above, but none 'slope away linearly from the BS'.
A straight line in a lin-log plot corresponds to an exponential term. The difference between two Gaussians of different width is probably another Gaussian. Are we learning that the real PSF has a Gaussian term in it that varies between filters? Since V-V images did not show this behaviour the Gaussian is not manifested by the inevitable slight image alignment problems. Our model PSF is an empirical core with power-law extensions - and the above experiments show that such PSFs do not yield linear-slope differences.
Perhaps we could study the real PSF by studying difference images in a thorough way? Student project
From flux to AlbedoPosted by Daddy-o Mar 21, 2013 10:49AM
In posts below we have discussed how to best investigate colour differences. Here
we saw that sky brightness and exposure time problems can be detected.
Using selected good images in B and V we found the pairs that also were close in time, and generated B-V images. We noted (discussed here
) that B-V on the BS is not always near the published value of 0.92, even in images selected for not having (obvious) exposure time problems. We wonder if the value 0.92 is more of a classical photometer value? That the colour of the whole BS on average is 0.92? Perhaps - but we also wonder if the reflectivity of the Moon has a phase dependence so that the B-V colour, even if a BS average, is lunar phase dependent?
Here, we choose to bring the B-V value of the BS in our selected images to 0.92 in order to study which values we get for B-V on the DS. This in an attempt to see if we can discuss what the B-V color of the DS, and therefore of Earth is.
For a range in lunar phases equivalent to illuminated fraction from 35% to 50% we plot the B-V values of a slice across the lunar disc, through the centre and 40 rows wide. We average over the rows:
We have aligned the images by the deep cut, which corresponds to the BS/DS border - the terminator. On the right of this we have the BS and on the left the DS.
We see that the BS is level. We have offset the image values so that the BSs are near 0.92 (by eye). We see that the DS has a slope. We see some level differences in these slopes but the slopes themselves are fairly similar. For one of the profiles the B-V reaches as low as 0.5ish, but there is still a slope.
On the basis of that I think we ought to say that "B-V for earthshine is at or below 0.5 in absolute terms". Better may be to say that "the DS B-V is 0.42 below the BS B-V level."
The B-V os sunlight is 0.656 ± 0.005 (Chris has measured a similar value of 0.64). If we also know that the sunlight on the Moon appears to us to have B-V of 0.92 then we can infer that one reflection from the Moon reddens the sunlight by (0.92-0.656)=0.264. The Sun also shines on Earth (lucky us!) and that light has its colour altered by the Earth - when it strikes the Moon and comes back to us it has B-V of 0.5 or less. Knowing what one reflection off the Moon does to the B-V color we infer that B-V of Earthshine is 0.25 or less. Franklin performed UBV photometry [with a 'diaphragm' of diameter near 1 arc minute - it means he observed areas on the DS, preventing the BS light to enter the photometer, but not BS-scattered light from optical elements before the photometer] on the Earthshine and found that B-V 'for Earth' was 0.17 below B-V for the Sun - this implies that B-V for Earth is 0.47. We have a value (from our lowest value) slightly above that.
1) The above slope is rather straight. The halo itself is there because the halo from B and V have not cancelled. This must be telling us something about the scattered light halos? Probably that it is linear in a lin-log plot - which is what we use using the 'BBSOlog' method. What would Ve1-VE2 images look like? We cannot tell since the halo is not (primarily) due to atmospheric scattering and Mette knows no rule for how scattering in lenses depends on wavelength. We have to try, before we know if Vegetation edge data can be extracted in this way.
2) We also see that we have no images without a B-V gradient across the DS. If we had had an SKE we might hope to achieve 'clean' B-V images. But we don't. So we can't.
3) All is not lost - when we fit images profiles directly we compensate for the halo and can therefore extract actual albedo data that does not depend on the presence of the halo.
From flux to AlbedoPosted by Daddy-o Mar 20, 2013 03:13PM
To check consistency we now look at V minus V images, where the two V images are chosen to be closer in time than 30 minutes but not taken at the same time. We correct for extinction; we convert the raw images (bias subtracted, of course) into instrumental magnitude images by calculating the flux from the nominal exposure times and taking -2.5*log10, and then align them and subtract. We shoul dget images of 0s since the fluxes, once corrected for extinction, should show the same flux - at least on the BS where the Sun is shining. We do this and get a royal mess:In each frame the insert shows a color-contour plot of the V-V image. The graph shows the usual slice across the middle of the image, averaging over 40 rows.
We see in upper left panel a clear offset at the terminator - i.e. the DS has different level but the BS are similar. Upper right shows a failry decent pair of images - the terminator is giving some problems and the DS as well as the BS differences are offset from 0 by a small-ish amount which coul dbe caused by an error in exposure time of some 10% or so (not unlikely). Lower left shows that while the two BSs are at teh same level then the DSs differ violently. The lower right shows a really nice example of two images agreeing.
What is going on? We hand-inspect the above images and see that in the case of the lower left image the sky level is much higher in one of the two images used, although they are observed less than 30 minutes apart. The counts inside mare Crisium are 3 and 11 in the two cases - i.e. a factor of about 4 or a magnitude difference of 1.5 - well outside the plot frame. So, from this we learn that we have a method to detect stable sky conditions! It also tells us that using images from different filters should be done with great care - even small differences in sky conditions will cause the DS to shoot off!
As it is, these images are merely bias subtracted - there is no individual correction for a 'pedestal' due to sky conditions. Luckily we can still use these images for albedo work in that we expressly fit a pedestal term!
What else can we learn? Well, any error in actual exposure time will influence the DS and the BS with the same factor - hence the magnitude differences plotted above will become offsets for both DS and BS - hence, upper right is consistent with 'wrong exposure time' in one of the images. Having assigned one filter and received another will have the same effect as an error in exposure time - if the DS and BS are altered by the same factor. I wonder if 'wrong filter' could camouflage as 'wrong exposure time'? Since DS and BS have different colors I doubt it - but we should investigate this.
So, three things learned:
1) Wrong exposure time will lift or depress DS and BS by same amount.
2) More sky brightness in one image than in the other - affects DS only.
3) Image pairs like the ones used for lower right panel, above, are probably both OK.
From flux to AlbedoPosted by Daddy-o Mar 20, 2013 11:09AM
I have located all 'good images' in B and V. That is, all B and V images, made from stacks of 100 images, that are on Chris' list of 'good images'. I have furthermore identified all pairs of B and V images that are taken less than 1 hour apart. For all these pairs I calculate a B-V image, by using Chris' calibration of flux against known standard stars. I allow for an extinction correction based on kB=0.15 and kV=0.10. I get airmasses from the Julian date of the image, and IDL software. I plot a colour contour plot as well as a 'slice' across each image. The slice is made up of an average of 40 rows centred around the row that goes through the centre of the B-V image. A total of about 33 image pairs have been plotted in this way. Here is a (large) pdf file containing the plots.
There are several strange things to see. Generally we get the profiles shown here
. That is - the BS is somewhat flat, while the DS slopes towards the DS sky. Oddities include images where the DS is as flat as the BS - just at another level. Is that 'extremely good nights'? I think not
. In some, the DS is much higher than the BS - that could be images in which the exposure time is incorrect due to shutter problems. Or filter-wheel problems! In many the BS is flat, but not near the 0.92 value we expect based on publications. For small differences with this 'canonical value' I think we could be talking about exposure time uncertainties. If we are at B-V=1.0 instead of at 0.92 we could have an 8-10% error in the exposure time. The times are short and it does not seem unlikely we have that big a problem. Sigh.
I will suggest that we 'correct the BS value' to 0.92 by simple shifting, and then consider how our DS B-V values look. With some 'deselection' of some obviously bad images we can perhaps arrive at a base set of good B-V images.
I think the result will be that few show any leveling off in the 'slope of B.V wrt distance to BS'.
We have few images at small lunar phase (i.e. small illuminated fraction). Chris has analyzed one good pair of B and V images at small phase, but it is not in my pipeline of good images - the automatically detected centre coordinates and radius are way off (because the sicle is so small that automatic methods do not work). Our new student Johanne is inspecting these images by hand and will adjust radius and centre coordinates and then we should have a few more images to play with at small phase. These may show us if we can get far enough away from the BS that the DS slope in B-V levels off and shows the true earthshine B-V color.
From flux to AlbedoPosted by Daddy-o Mar 18, 2013 03:47PM[Note added later: we have updated this posting using extinction corrections for B and V. kB=0.15 and kV=0.1 were used, and airmass=1.67. JD was near 2456016.82]
In preparing for the EGU 2013 conference, where we will have a poster on color results from the earthshine project, I want to show a 'B-V' image of the Moon - since we can, and may be the first to publish such a thing.
Using Chris' calibration of our filters against standard stars in M41 and NGC6633 I can reduce observed images in any filter to 'magnitude images' in the same filters. By taking the difference between the B and V image I thus arrive at a 'B-V image'. Here is a slice across the middle of that image:Second image added later: We now correct fore extinction:
Here, the BS is to the right of the 'cut' in the disk, and the DS is to the left. It appears that the BS is higher than the published value for 'moonshine' of 0.92 [see e.g. Allen, "Astrophysical Quantities", 4th ed, Table 12.14]. Added later: Even after correcting for extinction the B-V is still higher than Allen's value - we now get B-V near 0.95-1.0. Perhaps now, not so much larger than that value?
I am not quite sure about the DS. There is probably a slope because the halo in B and V are not identical and thus do not cancel. But as the BS is approached the color of the halo does approach the BS value, since that is where the light comes from. So I understand that part.
At the extreme left of the DS there is least influence from halo. The value is a bit up and down, but appears to be lower than the BS by up to 0.3. If B-V is a smaller number in one place compared to another it means that B is smaller than V in the first place, relative to the second - i.e. on the DS B is relatively smaller there than on the BS - since these are magnitudes it means we have shown that earthshine is blue - the Earth is indeed 'a blue marble'!
Here is the B-V image itself, with cleverly chosen colours to reinforce our message: Sky is masked out, BS is to the right, DS to the left.
I'll try to redo this for a lunar phase with less halo.
Data reduction issuesPosted by Daddy-o Mar 12, 2013 04:11PM
We have been following the mystery of the VE2 filter. Ana Ulla asked a good question: "Do you see the same problems in images not of the Moon?", so I checked that, using our copious collection of Altair images. I fit a Moffat profile to each well-exposed image of Altair and extracted the sky level. I plot it here as a function of which filter (0=B, 1=V, 2=VE1, 3=VE2 and 4=IRCUT) and as a function of time, for one night (JD 2455845):
In the top panel we see that VE2 (the brownish points) have two levels, and in the lower plot we see that the VE2 level switches from high to low after some time. (Bottom panel has fractional day as x-axis). We also see that the black points (B) increase their level (slightly) at the same point (0.87) where VE2 drops to the low level. Blue points (VE1) have some history like that too. V (red) is quite stable. Green is IRCUT.
If the excess in VE2 (and others) was due to a source of light, then we would hardly see a down-shift in the level. Ana's idea was that thin cirrus could enhance the nIR band of VE2 (quite like the Johnson I band) - but that does not explain why B rises, unless that is a particular behaviour of cirrus clouds, which I doubt. I do not think anything above is consistent with a light source. I think this is what we have had to conclude from the other results presented in this blog - e.g. the inverse exposure time dependence and the inverse airmass dependence.
I am guessing when I say that I think it is something to do with electronics. The camera knows which filter is being used! (Not
From flux to AlbedoPosted by Daddy-o Mar 11, 2013 09:53AM
We have already shown a strange behaviour of the VE2 filter - or what appears to be a problem related to the use of the VE2 filter. The post is here
Now we are able to report something even stranger (OK, just
as strange then). We appear to have a relationship between the 'VE2 pedestal' and the airmass of the observation:
In the upper frame we see the old VE2 pedestal vs exposure time plot. In the lower frame we see that the pedestal is also a function of the airmass! If the pedestal was proportional to airmass we could, perhaps, understand the phenomenon as something showing a flux proportional to the amount of atmosphere (I am thinking of some sort of airglow here) - but NO, the pedestal height is smaller
the larger the airmass is!
Is this some electronic problem related to how the telescope is oriented? Then why the inverse proportionality on exposure time too? Why not the other filters?
going on here??
From flux to AlbedoPosted by Daddy-o Feb 28, 2013 10:08AM
In this entry: http://iloapp.thejll.com/blog/earthshine?Home&post=304 we investigated the effects of using a lunar albedo map based on scaling the Clementine map to the older WIldey map vs. scaling the Clementine map so that lunar mare and highlands matched what is published in the literature. [Note: the point being that the 'Clementine map' we have available is just a picture file - jpeg! - so that pixel values have to be scaled to albedo values somehow.]
We were doing this fitting in the 'new way' which is to fit 'profiles' starting on the sky on the DS side and extending onto the DS itself, modelling the contribution from the scattered BS light. In doing this we saw that improved fits could be obtained if the lunar map albedo was 'stretched' so that dark and bright areas better matched our observations. We did this stretching by eye and were able to improve the formal fits.
We have now compared the fits on 535 images done using the 'Clementine scaled to Wildey' map and the 'Clementine stretched by eye' maps.
We have found that the RMSE (that is, the square root of the sum of the squared residuals [i.e. observed profile minus best fitted model profile]) is improved using the scale by eye' map - in two ways: a subset of images were poorly fit using the other map; they are now much better fit, and the mean RMSE of the finally selected images is lower.
We selected 'good fits' on the basis of alfa (the PSF-width parameter) having to be in a narrow range and that the relative uncertainty on the fitted albedo should be below a certain limit.
Mean log10(RMSE) is now near -1.1 in units of counts/pixel. We fit a profile that is 150 pixel columns long - 50 pixels on the sky and 100 on the DS.
Mean relative fit-uncertainty on the albedo is near 1.5% when using 'counting statistics errors' on the observation.
We note that the VE2 images more frequently have a larger 'pedestal' or sky offset after bias removal than the other filters. While most filters have an offset of near 0.2 counts (+/-0.5 or thereabouts) the VE2 offset is more often near 4 or 5 (+/- 1-2 counts). What is the cause of this? The bias frames surrounding the VE2 exposures have been spot-checked and seem OK - bias is near 400. A few, otherwise perfectly all right, VE2 images have a sky offset of 10-40 counts! Observations from the same night in other filters show nothing like this - some sort of nIR 'fog'? Does the sky emit nIR light?
Does this have a conseqeunce for the 'tunnel selection' of VE2 images, done by Chris? [A data-selection method designed to take into account shutter and filter-wheel problems by requiring that image total fluxes follow a known phase-curve.]
While the 'new fitting method using stretched lunar albedo maps' formally works best of all methods we have seen so far, it is very empirical. Later we may be able to use the method to produce a 'best fitting lunar albedo map' that improves on the original Clementine map. We may then also be able to compare to what the LRO people (and LLAMAS) are finding.
From flux to AlbedoPosted by Daddy-o Feb 25, 2013 08:49AM
After the scattered light has been removed we still have the task of converting the observations to a terrestrial albedo. This is done, by us and the BBSO, with the use of a model for how the Moon reflects light. This involves assuming a lunar albedo.
The BBSO has a set of lunar-eclipse observations that they use to find the DS/BS patch albedo ratio - we use the Clementine map. That map is available to us only as an image file. We scaled that image so that the lunar mare and highland albedos correspond to what the literature states.
An alternative would be to scale it to match what lunar albedo maps show. The only digital lunar albedo map we have is the 1970s Wildey map, which Tom Stone gave us. In this blog: http://iloapp.thejll.com/blog/earthshine?Home&post=253
we performed a regression-based scaling of all the pixel values in the Clementine map to corresponding (or interpolated) pixel values in the Wildey map, using the two maps' coordinate systems.
We therefore have the possibility of fitting our observations to the lunar map of Clementine scaled to Wildey (let us call that the "Wildey map" from now on) or the Clementine map ("Clementine map" - but note that both are Clementine-based!) scaled to the highlands and mares. This choice has been investigated.
We find that the contrast in the Wildey map is lower than in the Clementine map, which affects the quality of fit.Same lunar-edge profile (white line) fitted with two different models (red line) - one uses the Clementine lunar albedo map, scaled to match lunar highlands and mare; the other is the Clementine map scaled, using every pixel, to fit the Wildey lunar albedo map. Notice the smaller contrast between highs and lows in the upper image (Clementine scaled to Wildey) compared to the lower (Clementine scaled to highlands and mare).
There is evidently a difference in the quality of the fits - neither being particularly good, failing to match the observed highs and lows. The conseqeunces for the albedo fitted is at the 3% level.
The mean difference in the terrestrial albedo determined using the two maps:
16 albedo values determined using both Clementine-scaled-to-Wildey (vertical axis, in top plot) and Clementine alone (h. axis). We see that the two albedos are near the diagonal - with only a few outliers. The histogram shows the absolute difference between the two albedo determinations - it is about 1% of the albedo value.
As long as the large spread in albedo values (from 0.23 to 0.42) is real and not due to some data-treatment bias, we have an effect from the choice of lunar albedo map that is at the level of the errors from pixel statistics (shown elsewhere in this blog).
We would like the known biases to be less than the effects due to scatter, so perhaps some work on the lunar albedo is in order? We can look at the mean values of the two maps - there was no requirement that mean albedo be conserved, and we can look at various ways of stretching the contrast, while maintaining mean value - this is a nice piece of work for a student project.
Performing a hand-adjustment of the contrast scaling, while maintaining the original Clementine map mean, we can refit the relevant profile. We get this:
Smaller RMSE, small but important change in ftted albedo.
Showcase images and animationsPosted by Daddy-o Feb 21, 2013 09:17AM
During processing a strange signal was found in a frame combined from 100 images. It turns out that in one image of this stack we have what appears to be a meteor or a satellite flash - or something:
The image is from Jan 17 2012: (UTC 2012-01-17T13:28:48). The exposure time was 0.009 seconds! The trail is about 3/4 of the lunar diameter in length - i.e. about 22 arc minutes. The orientation is such that it is travelling almost Due North (or South!).
What is it? Well - it is clearly between us and the Moon! If it is a meteor its height would be something like 50-100 km. The speed would then be 35 - 70 km/s. A satellite in low earth orbit has speed 8 km/s. Since the image is taken from Hawaii at UTC 13:28 it is near midnight on Hawaii - i.e. the Sun is behind the Earth and unlikely to be illuminating a LEO satellite.
An airplane flies at 800 km/hr at altitude 10 km, so the distance covered in 0.009 s is 2 m which would subtend an angle of 1 arc minute. This is no airplane - or it is much closer, in which case we should see details of the plane.
As far as I can tell there is no pronounced peak in meteorite activity in January.
Data reduction issuesPosted by Daddy-o Feb 12, 2013 11:41AM
In several posts we have considered the behaviour of variance-over-mean images (VOM). In the ideal world these should be 1 because the noise (once RON is subtracted) is Poisson-distributed. We have seen how this is hardly the case in observed images.
We now consider a test using ideal models with and without imposed image drifts. We generate 100 synthetic images with Poisson noise, and RON as well as a bias.
Below, each set of three panels show a cut across one image in the synthetic stack, then a slice of the VOM image before image alignment and then the same slice after alignment. On the left are images that were not drifted while to the right images were allowed to 'jiggle around' by a few pixels.
We see that VOM is 1 on the BS in un-jiggled images, but that DS and sky values fall below 1.
We see a HUGE effect on images that were allowed to drift.
Some of it we understand. Before alignment
, VOM rises on structured areas of the drifted images because surface albedo variations are being introduced for a given pixel along the stack direction. The effect on the DS and sky is much less - perhaps because the Poisson noise is so large comapred to the signal variations. After alignment
, VOM falls to slightly below 1 on the BS, except near the edge. On the DS and the sky, though, a large lowering is seen. So far it is not understood how this comes about.
Any strange effects seen in observed images will be all the larger since the images do not just drift but also 'shimmy' because of atmospheric turbulence.
The effects of aligning images to sub-pixel shifts is part of the above.
Let us learn from this that a noise model probably is hopeless to build in the presence of image shifts - despite realignment - and that sub-pixel interpolation is not a welcome added bother. We could just omit single images with the most drift and use a straight average of the remaining stack. In real images we have the option to not use stacks that have a l ot of drift - but we do not know the extent of the 'shimmy' for the remainder.
These realizations above have the most impact on our ability to interpret results that discuss the effect of alignment - alignment reduces some problems but probably adds some others.
Data reduction issuesPosted by Daddy-o Feb 08, 2013 12:37PM
It would be a Good Thing if we could explain observed images in terms of a 'noise model' - that is, a model that explains why the signal and the noise bears a certain relationship.
We consider here a stack of 100 raw images taken in rapid succession. The difference between adjacent images should only contain noise in that any objects shown in both images should subtract. The difference between two series drawn from the same poisson distribution but independent has a variance that is twice the variance in either of the series subtracted, so we apply a factor of 1/2 to the difference image to get the variance image.
We subdivide such a variance image into smaller and smaller squares, and in each square we calculate the variance. We generate an image of the ratio between the variance image an dthe mean original image, also subdivided In the case where everything is Poissonian this image should be a surface with value 1. In reality there will be noise in this surface - and there will be strcutures seen wherever imperfect obejct subtraction took place. Here is the result:In the three rows we results from different subdivisions - into 8x8, 4x4 and 2x2 squares. In the leftmost column is a histogram of the values of the image and to the right is plot of the profile across the middle of the image.
We see, to the right, that variance over mean (VOM) is not 1 everywhere. We are evidently picking up a good deal of variance near lunar disc edges [The image corresponding to the above is a quarter Moon with the BS to the right and DS to the left, situated in the image field center]. We see that the sky manages to have VOM near 1 and that parts of the DS does this, but that most of the BS appears to have VOM>1. Even ignoring the peaks that are due to edges we see a value for VOM near 2 or more [also seen in the rest of the 100-image sequence].
Since we used raw images we have to subtract a bias level. We subtracted 390 from all raw image pixels. We applied the ADU factor of 3.8 e/ADU to all bias-subtracted image values. We then calculated the variance and subtracted the estimated RON (estimated elsewhere at near sigma_RON=2.18 counts, or variance_RON=4.75 counts²; consistent with ANDOR camera manual "8.3 electrons").
The subtracted bias is a little small - the observed mean bias value is nearer 397, but if we use that value we get strange effects in the images - only a relatively low value of the bias mean gives a 'flat profile' for the sky in a slice across the image. This is one poorly understood observation.
We also do not understand why the VOM is nicely near 1 on the DS while it fails to clearly be near 1 on the BS - both areas of the Moon have spatial structure which is bound to contribute to the variance in the difference image during slight misalignments.
That is the second poorly understood thing.
Progress towards a 'noise model' is therefore underway, but there is some distance to go still.
How would non-linearity in the CCD influence the expectations we have from Poisson statistics?
Data reduction issuesPosted by Daddy-o Feb 07, 2013 10:42AM
We are using stacks of 100 images. These are acquired by the CCD camera in rapid succession - 100 images can be obtained in less than a minute. During that time there may be small motions of the telescope, and the turbulence in the air, as well as the slight change in refraction due to airmass changes may cause image drifts in the stack.
In one approach we ignored the possible drift and just averaged the 100 images. In the other approach we use image alignment techniques to iteratively improve the alignment of the stack images: First we calculate the regular average, then we align all stack images against that average image, then we calculate a new average image on the basis of the alignments and re-align all images against this average and so on. This procedure turns out to converge, and after 3 iterations we can stop and save the last average image.
We would like to know the effect of doing this on result quality. We therefore generated two sets of averaged images - the first using the simple first kind above, and the second the iterative method.
We estimate 'errors' in the bootstrapping way. That is, we extract MHM averages (mean-half-median, as explained elsewhere) of the DS intensity on raw and cleaned-up image patches and also estimate the statistical error on these values by bootstrapping the pixels inside the patch, with replacement. This bootstrap procedure gives us a histogram og MHM values and the width of this distribution is a measure of the 'error on the mean'. We express this error as a percentage of the mean itself, for RAW images and images cleaned with the BBSO-lin, BBSO-log and EFM methods.
we now compare the results to see the 'effect' of performing alignment of stack images:
Table showing errors (in percent of the mean). Lines labelled 1: and 2: show results for 'one-alfa PSFs' with and without alignment. The third line shows the effect of using a 'two-alfa PSF' and alignment.
We see that there has been a large reduction in errors by using alignment. Raw images improved by 20%, EFM images by 17%, BBSO-lin by 16% and BBSO-log by 14%.
The effect of using a two-alfa PSF on aligned images is small - indeed, all images except EFM experience a small increase in the error (probably not significant).
The effect of alignment on single-alfa PSFs is not investigated.
We conclude that alignment is a beneficial operation. We note that EFM has the largest errors but other arguments imply EFM is the better method to use - this is related to the stronger phase-dependence seen in non-EFM images, and is discussed elsewhere.
Post-Obs scattered-light rem.Posted by Daddy-o Jan 22, 2013 01:39PM
We have some images of the SKE inserted in the ray path of our telescope, during Moon observations. We also have some images of the same from the BBSO telescope, kindly lent to us by Phil Goode.BBSO images om two separate periods about a week apart. Top row shows histogram equalized imges of the BS (left and the DS (right) with SKE superimposed over the BS. Lower row shows the same - BS short exposure on left and DS long exposure with SKE on right.Histogram-equalized image of the Moon with SKE inserted from our own telescope. SKE runs from upper left to lower right and is opaque above that line.
Our image shows a halo around the Moon that stretches behind the SKE - hence we know that halo is formed to a large degree in the secondary optics! On the BBSO images we do not see this as clearly - either because the BBSO telescope is of a different design or because the exposures with SKE (right column of images) covers much more of the BS. Assuming the latter we understand why there is no halo behind the BBSO SKE - the BS was effectively blocked and no bright light entered the secondary optics and could not cause a halo. On our image the SKE is inserted experimentally only, leaving a large part of the BS to shine into the secondary optics.
So, we learn that the halo is not from atmosphere or primary optics alone - apparently a large fraction of it comes from the secondary optics!
We also see the importance of having an SKE. While the BBSO group uses the 'linear BBSO method' to remove scattered light over the DS they have a much smaller problem than us because the halo from the BS is nowhere near as strong as ours is!
We now see how terribly important the SKE is.
From flux to AlbedoPosted by Daddy-o Jan 22, 2013 09:09AM
Following on from post
we now inspect satellite images one week apart in order to understand the natural variability found in satellite images of the same area.Upper panel: average image pixel value for sequences of MTSAT images, 1 hour apart, for almost one day in March 2013 and a week later (red curve). Image pixel values is in arbitrary units but is proportional to pixel brightness. Lower panel: difference between upper panel black and read curves, expressed as a percentage of the mean of ther ed and black curves.
The difference between the two curves is on the order of 10% and varies from 8% to 18% during one day.
This tells us two things: Albedo (or something proportional to albedo) can vary by roughly 10% over a week. Albedo can vary during one day by almost as much.
This is useful information to have when we interpret the earthshine data.
We keep in mind that the smooth variations in the black and red curves in panel one are due to the day/night cycle - not intrinsic albedo variations: but the difference
between the curves and the variability in the difference tells us about albedo variations.
From flux to AlbedoPosted by Daddy-o Jan 21, 2013 09:35AM
We can reduce our observations to a number that is equal to the 'Lambertian albedo of Earth' at the moment of observation. That is, for a sphere behaving like a Lambertian sphere we can find the single-scattering albedo that gives the same earthshine intensity as we observe. This takes phases and all relevant time-dependent distances into account.
In order to understand if the data we get are realistic we wish to compare to satellite images of the Earth. From work published by others [the Bender paper] we are told that the terrestrial albedo varies by many percent from pentad [avg, over 5 days] to pentad.
Many of the best observations we have have 'sunglint coordinates' over South-East Asia. One geostationary satellite hanging over that spot is the Japaneese 'MTSAT'. We may be able to get data from it, but have only ready access to a Meteosat that hangs over the Indian Ocean.
From the Indian Ocean Meteosat we have extracted a series of images for a given day, in order to start to understand what sort of variability we shall expect on time-scales that are shorter than a pentad.
We have access to half-hourly images from the satellite and have 12 hours of data - 25 images. We take the average of each image and plot it:
The black line is the observed mean intensity of the whole-disc image, and the red curve is a 6th order fitted polynomial. The difference between the black and the red curve has a standard deviation of 0.6% of the mean of the black curve.
The large-scale behavior of the black curve is due to the phase - we see half a day pass as seen from the satellite so Earth changes phase from new to full to new again. Variations over and above that would be due to changes in earth's reflectivity. There is a pronounced sunglint in the Arabian Sea near Noon.
Some pixels in the image are saturated.
What do we learn from this?
By removing the fitted curve we learn how much variability there is as a day passes. Some of this would be due to the curve not actually being a 6th order polynomial - so the variability we get from the residuals are an upper limit. Fitting a 7th order polynomial lowers the S.D. to 0.4% of the mean.
It seems that the presence of a sunglint for some of the frames (the sunglint moves on to land - Horn of Africa - in the local afternoon and does not 'glint' in the sea anymore) does not generate a 'spike' of any kind in the average brightness.
We should remember:
That average brightness is not the same as albedo since the brightness depends on albedo times a reflectance function. But most of the reflectance behavior is removed via the polynomial fit, we think.
That cloud-patterns on this scale hardly vary much.
We do not yet fully understand the processing of the satellite images - are they normalized somehow? Were they all taken with the same exposure time?
On this day at least variability in the albedo was much less than 1%.
We should inspect several days of images to see if the average level differs much from day to day.
We should try to get information from MTSAT, and at higher cadence - we can observe the earthshine with minute spacing.
Try to get more technical information about the satellite images.
Post-Obs scattered-light rem.Posted by Daddy-o Dec 19, 2012 02:07PM
We do have many problems to contend with - but now and then we are confirmed in what we set out to do: Here is an example of the scattered-light reduction of an observed image (2456034.114etc) where the EFM Method seems to be doing well compared to BBSO linear. We compare to a synthetic model image generated for the moment of observation (i.e. the libration and geometric factors are representative of the observing situation):Top panel: slice across the Moon at row 296 so that Grimaldi near the edge is transected. The black curve is the synthetic model unconvolved - it is in units of W/m². The red curve is the scaled and offset profile along the same row of the EFM-cleaned image; and the blue curve is the BBSO-linear cleaned image identically scaled and offset.
Middle panel: detail of the above.
Bottom panel: difference between EFM and ideal and BBSO-linear and ideal, along row 296, expressed in percent of the ideal value.
The observed data were scaled since the units are different; they were offset because the EFM model did not have 0 value on the DS sky. The BBSO-linear, being 'anchored in the DS sky' did have a 0 value in the sky. The same scaling factor and offset was used on EFM and BBSO, for comparison, however - hence the little offset on the DS sky at right.
What do we see? Well, the EFM-cleaned (red) line follows the synthetic model quite nicely between column numbers 325 and 370 while the blue line (BBSO-cleaned) diverges all the time. Near the DS edge (columns 370-385) the synthetic model is higher than EFM.
What does it mean? The BBSO-linear method has better removed the flux on the sky - it is designed to do that, while the EFM is designed to minimize the residuals squared in a mask on the sky around the Moon. This implies that the BBSO-linear method, in the present case, would be better than EFM on the lunar disc near the DS sky. As we move further onto the DS disc the BBSO-linear method will fail more and more since the halo is not linear with distance from the edge - the method underestimates the amount of scattered light on the disc between the edge and the BS. We do see how the blue line diverges more and more; we do see the red line cling closer to the 'true value' (assuming the synthetic model is 'true') across the disc, before it too fails nearer the halo and the BS. The behavior nearer the edge may be a consequence of how we model the synthetic images - we have to use a reflectance model to make the synthetic images - and the angles of emergence and incidence corresponding to 'near the edge' is not one for which the reflectance model is inherently very good. The models we use are based on the simple 'Hapke 63' model. We speculate that the disparity between observations and synthetic models in the columns 370-385 is due to model inadequacy.
Anyone using a reflectance model as simple as the Hapke 63 model will encounter the above problem if they try to use pixels near the lunar disc edge - the natural thing to do, if the halo is removed using the linear method is to use edge or near-edge pixels.
Hence the problem of reflectance modeling and the inadequate linear method become coupled! Our EFM method seems to be a way around this obstacle - allowing use of disc areas further from the edge where simple reflectance modelling is adequate - hence we should be getting more reliable terrestrial albedo data. One day.
Post-Obs scattered-light rem.Posted by Daddy-o Dec 19, 2012 08:50AM
In the EFM-method we determine the alfa values for every image. Is there a link between the alfa value for one filter and the rest in an interval of time? It is our understanding that alfa is determined by the amount of scattering in the optics plus the atmosphere. We therefore expect that on 'bad' nights the alfa values will tend to move in the same direction. We investigate this here.
We find all alfa values in all EFM-treated images. We sort them into half-hour bins. We calculate all the alfa values in each bin and plot the results. Below is a pdf showing all the plots between some filters, at different 'zoom-levels'. The image shows the last zoom-level, highlighting the dense 'clump' of points:
There seems to be a general agreement that the alfa values are correlated - bad nights (i.e. 'broad PSFs') occur in all filters at the same time. Since VE1 is just about identical to IRCUT the scatter seen above means that the fitting routine is unable to make a perfect match - or that observing conditions, during the half-hour bins used, changed.
Using 15 minute bins does not improve matters:
I therefore suspect that the fitting method does not find the best fit each time.
From flux to AlbedoPosted by Daddy-o Dec 17, 2012 02:19PM
What B-V colour should we expect for the earthshine?
We will here estimate it by using the change in colour of Sunlight that has struck the Moon once, and the colour of Earth as estimated from spacecraft.
The Sun's B-V is +0.650 [Allen, 1973] [Holmberg et al, MNRAS, 367, 449, 2006
The Full Moon's B-V is +0.85
[Lane&Irvine, AJ 1976 78, p. 267]
[vdBergh has +0.876
for Mare Serenitatis;
Allen 4.ed. table 12.16 has 'Moon' B-V 0.92
Gallouet (1963) has +0.94
Wildey & Pohn (1964), AJ vol 69, p.619 have a range of values near +0.86 to +0.87
(their work seems good and a milestone).]
The Earth's B-V is 0.2 [Allen 3. ed, but appears based on a 1961 work - so pre-spaceage?]
The Moon's DS B-V is 0.64 on average given data in [Franklin (1967), JGR 72, p 2963]
If Sunlight is reddened by one reflection off the Moon by 0.85-0.65=+.2 mags, then we expect earthshine, bounced once off the Moon to redden by the same amount.
If the Earth has B-V=0.2 as seen from space then seen after one reflection it ought to be redder by +0.2 or appear to us observing it from Earth at B-V=0.4. This is not what Franklin measured.
Basically, we do not yet know Earth's B-V colour! I am making inquiries, and we shall see.
Note that Danjon did lots of colour observations of earthshine - but in the Rougier system. Wildey [JGR vol 69, p.4661+] refers to a transformation from Colour Index ("C.I.") in the Rougier system to B-V in the Johnson system - but without giving numerical details. The method is based on transformations using the Full Moon and the Sun colours.
The transformation should be made specific and the data from Danjon placed online. Another student project!
Post-Obs scattered-light rem.Posted by Daddy-o Dec 17, 2012 11:34AM
We compare the B-V values on the DS of images from JD2456034: we look at images only exposed to bias-removal ('DCR' images) and images cleaned with EFM and images cleaned with the two variants of the BBSO method: linear and log. We show a 'slice' across the disc at row 256:Top left: black is the B-V slice from the DCR images; red is the B-V values from the BBSO-linear cleaned images. The second graph in the upper left panel is the difference between the B-V values (BBSO-linear images minus cleaned image). The vertical dashed lines show the edge of the lunar disc and the start of the BS in column units.
Top right: same, but for BBSO-log method.
Bottom left: same but for EFM images, as shown elsewhere on this blog.
The results are quite different - the BBSO-linear method has given us fairly constant B-V values across the disc - they are about .05 mag below the DCR values (i.e BBSO-linear are bluer than DCR values).
The BBSO-log values seem completely unrealistic.
The EFM-cleaned values also look a bit unrealistic in that there is a spatial dependence on the magnitude of the B-V relative to DCR - in a way that looks like a remnant effect of the halos. The Delta(B-V) value changes sign across the disc.
The night JD2456034 is very close to New Moon and we know from other results that this is when the BBSO-linear method is likely to work best (the halo being small). Since there is a phase-dependency in the overall results for BBSO methods over and above what EFM shows we know we cannot universally use BBSO results. On the DS, towards the edge, the BBSO-linear method should be very good - it is 'anchored in the sky', unlike the present EFM method, and therefore should be unbiased near the edge. Our EFM method, at the moment, only minimizes the square of the residuals on the sky, inside a mask.
Near the sky, the DS B-V values in the BBSO-linear and EFM-cleaned images differ by about 0.04 mags. We should look at EFM methods that also 'anchor in the sky' and see what we get then.
The present largest worry is not the B-V offset but rather the dependence on position on the disc that the EFM method shows.
What does the literature tell us we should expect B-V to be under earthshine?
At the moment I only know of Franklin's 1967 paper. It gives B and V values of the earthshine.http://adsabs.harvard.edu/abs/1967JGR....72.2963F
The average B-V seems to be +0.64. Can we find any other information on B and V?
Post-Obs scattered-light rem.Posted by Daddy-o Dec 10, 2012 04:31PM
In this entry on the blog: http://iloapp.thejll.com/blog/earthshine?Home&post=272
we discussed the Laplacian method of estimating earthshine intensity by removing the influence of the scattered light halo. We have tested it on a good image from near New Moon (i.e. lots of bright DS, very little interfering BS). Consider this plot:
In the top two rows we see the observed image, the EFM-image and the Laplacians of these. In the second row second plot there are three additional curves, in black. The centre one shows the estimate of where the disc edge is, based on the estimates we have of disc centre and disc radius. The other two lines are parallel outliers at + and - 40 pixels from the edge. In the third row we show first the coaddition of all rows along the edge, using the estimates of the edge and the outliers as start and stop - this allows a coaddition that centres on the disc edge. Second in third row is the same thing based on the Laplacian of the observed image. We see the signature of the double derivative of an edge. In the last row we show the same as in the third row but for the EFM-cleaned image and its Laplacian.
The application of the EFM method removes most of the halo and this is consistent with the slight lowering of the slope after the 'step' in the first panel of fourth row, compared to first panel of third row. We extract the size of the 'jump' as well as the distance between the minimum and the maximum of the Laplacian edge signatures. The jump size is estimated as the difference between the mean of the first third of the curve and the last third. We collect the four estimates of the earthshine intensity - two jump heights and two estimates based on the Laplacian signature, and plot them below for 4 differently rebinned versions of the original images - unbinned; binned 2x2; binned 4x4 and binned 8x8. This is done to 'sharpen' the edge in case the edge is 'fuzzy' so that the 3-point numerical operator used in constructing the Laplacian will interact correctly with the edge.
We see color-coded graphs. The estimates of earthshine intensity based on the observed image and the EFM-cleaned image are black and blue, respectively. The estimates based on the 'signature size' in the Laplacian of the observed and EFM-cleaned images are red and orange, respectively.
The differences are quite large - several percent, and there is a dependence on the rebinning of the image - even in the case of the step method. The results that 'ought' to be the best is the jump-size one based on the EFM-cleaned one since the halo is absent and because the jump size method should have less statistical error-of-the-mean than the signature size method since more pixels are involved in estimating the former. The error-in-the-mean for the jump estimates for the unbinned image is of the order of tenths of percent - clearly dominated by systematic effects.
The above suggests that there are large method dependencies.
The 'jump estimator' may be victim of the slopes seen near the jump - especially in the EFM case - with such slopes the mean before and after the jump is dependent on just how you estimate the jump. Possible improvements is to extend a linear regression from the first half or third of the points, before the jump to the midpoint and do the same from the other side and calculate the 'jump size' as the difference between the two extrapolations. The slope is still a concern, however. It is due to the lunar surface albedo variations. The Jump size itself is a function of earthshine intensity as well as lunar surface albedo and therefore on libration, as well as the geometric distance factors. Being estimated right at the edge the reflectance is a constant, leaving lunar albedo as a variable factor (through libration). This could be taken out with knowledge of the parts of the surface of the Moon right at the edge - i.e. via modelling.
The 'signature size' method has different challenges - it is based on fewer pixels for one thing. It also depends on the lunar albedo - and not on reflectance, provided the edge is used (and not features on the DS, as Langford et al do). However, the method depends on knowing that the image is in good focus. Bad focus will blur the edge causing a lowering of the signature size - for area methods such as the EFM we do not depend on a sharp edge but can estimate levels from regions chosen away from the edge. Of course, we still get blurring from lunar features overlapping.
The above study did reveal dependence on binning - but if blurring was being addressed the results are not encouraged as there was no convergence.
So many things to compare! Super student project! A study of artificially blurred synthetic images could be done, and 'step' and 'signature' methods compared.
Post-Obs scattered-light rem.Posted by Daddy-o Dec 10, 2012 11:29AM
In an interesting article Sally Langford, et al. [ http://adsabs.harvard.edu/abs/2009AsBio...9..305L
] describe an analysis method for earthshine images. Essentially, the Laplacian's effect on images is to detect edges and the size of the 'jump' in going from e.g. sky to lunar disc in the observed image translates into an amplitude of a characteristic signature of the derivatives of an edge. This enables the large-spatial-scale features of the scattered light halo to be removed from the short-spatial-scale features of edges and craters.
Our method for removing effects of halo is to construct a forward model of the halo, based on the BS of the Moon (caught in the same image as the DS - Sally's FOV is smaller - about 20 arc minutes; our is 60 arc minutes), and subtracting it, leaving the DS almost halo-free.
How well does the Laplacian method work on our type of images? Sally's images were well-exposed images of the DS alone - allowing large counts - 10's of thousands; ours are co-addition of 100 images taken so that the BS is not overexposed, leaving the signal on the DS typically near 10 counts. One image with 10,000 counts should thus have the SNR of 100 of ours.
We took one of our good images - from JD 2456095 - near New Moon - and plot the profile across the image near disc centre in the image that we obtain using the forward modelling method (named 'EFM' in the plot and this blog) and the same profile from the Laplacian of the image. We average 20 rows.
Inspection of the columns (approx # 100-120) where the lunar disc edge is we see a clear 'step' up from the sky level in the EFM-cleaned image but there is no sign, above the noise, in the Laplacian of the same image. Inspecting the whole Laplacian of the EFM image we do see a faint signature:
The above image is 'histogram equalized' to show the feature. Our own method - also shown when histogram-equalized is here:
We see that the halo has been only partially removed on the right.
I think we would be hard pressed to extract a signal from the Laplacian of the image. It also bothers me that the whole earthshine signal is reduced to the value of the derivatives in just edge pixels. In our own image we have hundreds of pixels to measure on - in the Laplacian we get only a signal along the edge.
We should note that the remnants of halo on the right are much less evident in the Laplacian image, suggesting that there is a kernel of a good idea in the method. For reference, the Laplacian of the raw observed image is here (histogram equalized):
It does seem as if the Laplacian helps remove a lot of the halo, but also reduces the analyzable part of the image to the DS edge.
The Langford paper mentions both smoothing images and co-adding them - the former is done at the resolution of the worst image for a sequence. The Langford paper analyses features on the disk - not the edge.
Working out how the Laplacian is best used, on realistic images of the type we have, would be a good student project!
Post-Obs scattered-light rem.Posted by Daddy-o Dec 04, 2012 04:23PM
We have, below, studied the 'structures' or 'bands' that, in some images, stretch from side to side. One concern was that the structure was induced by some step in image reductions. We therefore compare raw images to EFM-cleaned images to see if the structure is present in both.
Visual inspection of histogram-equalized versions of raw and EFM versions of the images reveals the structure, although it is easier to see in the EFM image since the halo and most of the BS is gone:
Plotting the average of the first 20 columns of each image, and scaling them to each other shows that the structure better:
The black line is from the EFM-cleaned image, while the red curve is from the raw image.
This seems to rule out that the 'structures' are induced by image-reduction steps.
We remain suspicious of the possibility that the bright light from the BS itself somehow is causing the problem. It is not a reflection since the structure is a deficiency of counts. Only way to explain it is by some process subtracting counts from these bands - or lowering the sensitivity of the whole set of bands - perhaps some sort of non-linearity caused by strong light in one part of the CCD affecting the whole row?
We still need to map when the structures is present - we want to see if it appears in a timeline or is a function of the placement of the image in the frame. Inspecting the first 700 images of 2000 for the EFM frames shows no sudden dependency on time of this problem.
Idea: also extract the maximum counts of the image and use this as a parameter - we do have suspicions that non-linearity sets in far below 55.000 counts.
Question: Why is the structure along rows, while readout direction is along columns? What is special, in a CCD chip, about the rows?
More to follow.
Bias and Flat fieldsPosted by Daddy-o Nov 30, 2012 02:43PM
In the entry below, at:http://iloapp.thejll.com/blog/earthshine?Home&post=269
we considered the need to remove some residuals left over when a poorly-fitted halo had been removed from the observed image. Sensing that the problem has to do with asymmetry in the solution forced by the lunar disc not being well centred, we consider now the effect of lunar position in the image on the quality of the halo fit. We estimate the quality of the halo fit from the mean value in a sky-patch near the DS.
We see here on the x-axis the value of the disc centre coordinate (i.e. column number in the image) and on the y-axis the mean value of the DS sky patch. We seem to have some scatter as well as a structure that looks like an inverted parabola, for these points near y-value 0. That is - the sky-patch mean value of the DS in images where the halo has been removed with the present EFM method depends on where the lunar disc is - the further away from x0=256 (middle of image) the Moon is, the larger a residual is left on the sky after the halo is removed with the EFM method.
We need to invent a better EFM method!
Bias and Flat fieldsPosted by Daddy-o Nov 30, 2012 08:45AM
In this post:http://iloapp.thejll.com/blog/earthshine?Home&post=268
I pointed at the unwanted structure in the sky near the lunar disc in an image that was Bias-reduced as well as had had its halo removed. Bias problems were ruled out - and left was a speculation on internal reflections in the camera system.
If we are truly left with that horrible structure on the sky we cannot just remove the halo in the present way, as the method depends on 'fitting the sky'. If there are basins and hills in the 'sky' the fit will be bad - if the halo subtracted has the right shape but not the right 'level' then perhaps we can think of a fix: After removing the halo, but before extracting photometry from the DS disc in the image we can reference the sky adjacent to the DS and offset our disc value from there.
I tried this, by estimating the sky level on a part of the sky near the DS on an image where the halo has been subtracted, following the EFM method:
The two patches on the lunar disc are the Grimaldi and Crisium reference patches (DS and BS, respectively [although the BS value is estimated from the same pixels in the raw image]) and the large semi-rectangle on the sky next to the DS is the 'skypatch reference area'. We calculate the average value of that patch and subtract it from the value extracted for the DS. If the error is of an offset type, rather than a slope type we have then corrected for the halo misfit.
To see the effect of this problem we estimated the DS/BS ratio in all EFM-cleaned images, for each filter, with and without referencing to the sky level:
As usual, the jpegs above are poor in quality so we also post the pdfs:
Each panel contains three columns - the rows are for filters. The first column is the DS/BS ratio as function of lunar phase (Full Moon is at phase 0). The second column is the ratio of the DS and the total RAW image flux [for reasons explained elsewhere!]. The third column shows the 'alfa' value derived by the EFM method.
Almost the same list of images were used for the two plots above - note the differences by inspecting the alfa plots - crosses are present in one plot but not the other one.
We see that the sequences of points for the case 'skyptach reference level removed' are slightly fuzzier than the sequence where the reference is not subtracted! Before we can interpret this I think we need to restrict the two plots to the same set of images - I shall do this and return ...
[later]: This has now been done. For the IRCUT filter (341 data points) we show the effect of skypatch-referencing on the DS/BS ratio expressed as obs/model:
First we notice the wide span in DS/BS along both axes - this is not new: this is mainly the phase-dependence we see - this is MUCH LESS than would be seen for BBSO-method treated images, and is not the point - the point is that the spread along y for a given value of x - say near x=1 is something like 50%: That is - the effect of removing the sky-level from the DS value, in EFM-cleaned images, is to alter the DS/BS ratio by 50%. This means that the EFM-method did not do a very good job of removing the halo. It did remove a lot of the halo - but clearly not so much that a considerable effect is felt when the small remaining offset is removed.
This is important for our Science Goals: On the one hand we see the EFM method work much better than 'BBS linear' and 'BBSO log' methods in removing the effect of phase on the DS/BS ratio [shown elsewhere in this blog], on the other hand we see it does not do a sufficient job.
We should note that the BBSO linear method removes the sky level in one step (since it is a fit to the sky near the DS) but that, being linear, it does a poor job of removing the halo where it matters most - on the DS disc. The EFM method may be much better at removing a halo of approximately the right shape - but there remains a bias that is important.
The current EFM method is not 'anchored in the sky' - it merely seeks to minimize the residuals formed when an empirically generated halo is subtracted from the observed image - as evaluated on a masked section of the sky part of the image. There are choices made when that mask is set up, and they are:
a) The mask used above was such that sky was included on both sides of the lunar disc, set off from the disc by some 20 pixels and curtailed vertically at 0.7 radii (so that a 'band across the lunar disc, but not including the disc' is set up). This gave weight to both the part of the halo on the BS and the DS.
b) The mask is quite large to give access to a large number of pixels so that the effect of noise on the fitting-procedure was reduced.
The requirement in b) forces the choice in a) to be so large that any unevennesses in the sky (such as we suspect occurs due to internal reflections when the Moon is near the side of the image) has an influence on the quality of the fit, forcing the above consideration of a 'special removal of the sky level near the DS'. To this comes the problems of allowing the BS halo to influence the fit.
We should investigate how we can improve the EFM method. Possibilities include
1) spatially weighting pixels so that 'difficult areas' are avoided or are given less influence on the final fit - such as the bothersome BS part of the halo.
2) use much smaller areas to fit the halo on the DS to - such as a patch near the DS patch on the lunar disc. This has to be balanced - on the one hand we get more influence of noise (which influences the BBSO method too, as it uses 'narrow radial conical segments' on the sky), but on the other hand we eliminate the effect of the BS halo as well as the 'unevennesses in the sky field'.
3) One hybrid form of the EFM, tested earlier, consisted of enforcing that the model halo should be flux conserving (as now) and at the same time pass through an average point on the sky near the DS in the observed image.
We should implement a method that does not show the above sensitivity to removal of a sky reference level calculated on the already cleaned-up image. Access to a system without these internal reflections would also be nice. One significant step to take before that could be to find all images without the 'streak' effect that lead to the above considerations.
Bias and Flat fieldsPosted by Daddy-o Nov 27, 2012 03:25PM
A strange structure has been found in some reduced images. To the right we show a dark frame taken seconds before the image on the left. The image on the left is an EFM image - i.e. the halo from the BS has been subtracted after fitting to the sky areas of the image. We clearly see the structure in the left image - a 'band' stretching to the left of the DS. This structure is not present in the dark image - so it is not an artefact of bias subtraction [Of course - the bias subtraction is not performed with adjacent dark frames - noise would be added that way - rather, a scaled almost noise-free superbias is subtracted. It is like a lower-noise replica of the image on the right - i.e. no structure, just mild level of noise.]. Below the two we show a plot along a vertical axis in the two images: columns 50 to 150 were averaged and shown in the plot as the black and the red lines.
There is a very clear structure in the sky of the Moon image. Given that it is not induced by bias-subtraction it must be due to the presence of the Moon itself. We speculate that:
a) it is some optical effect - reflection - from the inside of the camera or telescope. Halo-subtraction has reduced the sky level to almost zero but a little too much has been subtracted in the 'dark band' and a little too little has been removed outside the band. The fitting of a rather smooth 'halo' from the observed image could give this effect, if the structure itself is present in the image.
b) some electronic effect is causing the rows with the very bright BS in to somehow 'jump low' due to some effect we do not understand. NB: The readout direction is 'down' - not to the left!
Note the presence of what looks like a truncated halo to the left in the image, at the frame edge: this could be an internal reflection showing the right hand side of the halo being reflected inside the camera. We saw this same effect in the animated sequence of the tau Tauri occultation - as the Moon migrated to the RHS of the image frame an 'echo' appeared on the left.
This therefore seems to support idea a) above. If this is the case we may have an effect on solution-quality from position of Moon in the image. We should investigate if there is a position nearer the centre that eliminates the effect, and then omit images that are too close to the edge.
Data reduction issuesPosted by Daddy-o Nov 23, 2012 09:06AM
One of the reduction steps performed has to do with the scaling of the bias images due to the thermostat-induced temperature variations in the CCD chip. This temperature variation causes a 20 minute period in the mean level of the bias with an amplitude of almost 1 count - thus of importance to our attempts to analyse extremely small signal levels.
We take Bias frames on both sides of all science exposures - one just before and one just after. If we were to just subtract the average of these frames from our science image we would be adding noise to the result - we therefore need to subtract a smoother bias frame. We have constructed a 'super bias' frame as the average of hundreds of bias frames - it is very smooth, but probably has a level that is unrelated to the actual level in each frame.
By scaling the super bias to the mean level of the average bias frames taken at the time of the science frame we get a scaled superbias that we can subtract - it has the right level and very little noise.
We need to understand how the scaling procedure performs, so we have extracted the scaling factor from the 5000+ exposures we have.
Top frame shows the factor on the superbias as a function of the sequence number, and the bottom panel as a a function of the Julian Day of the observation.
Most factors are near 1 but some stand out. 9 files have factors above 1.09 - their Julian days (integer part) are:
2455938 (7 images) 2455940 2456032 (one on each).
A list of the 209 images with factors over 1.08 is here:
The 4 unique JDs are: 2455814 2455938 2455940 2456032, with the majority of cases on 2455814.
These images should perhaps be inspected very carefully for problems with bias.
A close-up of the factors nearest 1 looks like this:
Data reduction issuesPosted by Daddy-o Nov 22, 2012 10:28AM
Using the methods described by Chris in the entry below - i.e. at http://iloapp.thejll.com/blog/earthshine?Home&post=265
I have also selected for the likely good data by taking extinction into account and looking for linear sequences. This resulted in a set of images that I deem 'good'. That list can be compared to Chris' list.
Chris has 3162 'good' images, while I have 2990. The cross between the lists finds 2273 instances on both lists. This list of 'jointly agreed good images' is here:
Methods for selecting good images differ slightly: Chris does not yet consider the alpha value found in the EFM method - I select for alphas in a narrow range near the mode of the distribution for each filter.
With converging selection criteria the list above would expand somewhat - perhaps best to keep the criteria a little different to avoid duplication of potential errors.
The joint list contains 49 unique nights. The distribution of images over nights is:
where the first column gives the number for the night. Note that towards the end of the sequence there are few images per night - that is because we were realizing the necessity to observe 'stacks' rather than sequences of single images which results in fewer co-added images.
A histogram of these data is shown above.
From flux to AlbedoPosted by Daddy-o Nov 05, 2012 01:14PM
The earthshine comes from the various parts of the Earth that are turned towards the Moon, and the Sun - all the clouds and oceans and deserts and ice-caps that are illuminated and visible from the Moon contribute to the Earthshine. Which parts contribute most?
We take a representative image of Earth, as seen from space, and investigate where the flux mainly originates.
Splitting the above JPG image into the R, G and B channels we can analyses where e.g. 10, 50 and 90 % of the light comes from. That is - we seek the pixels that contribute these fractions of the total flux, and identify them in images. Note that R,G and B refers to other wavelength intervals than the B, V VE1 VE2 and IRCUT bands we have - our B band is bluer than the 'B' used in JPG images.In the three frames we see 3 rows (B, G and R, from the top) - on the left in each panel is the original R,G or B band image, while to the right are the pixels contributing to the 10, 50 and 90 percentiles of the total flux in the image. The order of the panels is: top left 90%, left bottom: 50% and top right is 10%.
In the 90% images at top left we note that the B image (top row) looks different from the R and G images - the light in the B band comes from atmospheric scattering - Rayleigh scattering, and aerosol scattering - as well as the ocean and the clouds; other bands have more of their flux coming from clouds.
Variations in the blue may therefore tell us more about the atmospheric state than do the other, redder, bands. The Rayleigh scattering is due to molecular scattering - as long as the composition of the atmosphere is the same this ought to be constant in time; but some of the blue scattering is also due to aerosols and thus we may have a tool to investigate variations in the aerosol load. The longer-wavelength bands will tell us more about the continents. All bands are quite dominated by clouds - a small cloud can reflect as much light as a larger un-varying continental area.
The above is repeated here on another image of the Earth - more realistic as it is half-Earth. Image from Apollo 8.
And here is the B,G,R images and the 90% percentiles:Top to Bottom: B,G,R, Left: R,B or G-band image - right: 90th percentile image.
We again see that the light contributing to the blue image (top) is more diffusely distributed than in e.g. the red (bottom) case where most of the light comes from variable features like clouds. This implies that we should expect larger variability in our albedo data for the red images than the blue image.
Real World ProblemsPosted by Daddy-o Oct 23, 2012 08:38AM
What is the effect on observational coverage if we have different numbers of observatories and observe in different ways?
Since this depends on what we mean by 'observational coverage' we define OC as 'largest fraction of the time with continuous observations'. Note that this is different from, say, 'largest fraction of days where at least one good observation was made', OK?
For 1,2,3 ... observatories chosen from the list of known observatories (in the IDL code observatory.pro) and evaluating at 15 minute intervals we get the following (non-optimal, but pretty good) results when January and July are combined:Upper panel: The red curve shows OC for Moon above 2 airmasses and Sun lower than 5 degrees under horizon. Blue is same but for 2 airmasses. The dip at 5 in the red curve is an artifact of the search method we use - exhaustive search between 44 available observatories would be too expensive so we seach for best of 100 random picks of 1,2,3,4 ... in the list of 44. Lower panel: The same, but evaluated for more stations, best of 200, and with a 10% random - uncorrelated - occurence rate of 'bad nights' (clouds, for instance).
We see that, compared to this
we have less OC - that's because that search was for 'Moon above horizon, Sun below' instead of the more realistic constraints used here.
We see that extending observations from AM 2 to AM 3 is equivalent to adding two observatories for midrange values.
We see that adding many more observatories is in the end a loosing proposition - the 7th observatory on the blue line adds nothing compared to the sixth.
a)More exhaustive searches can be made, but takes time. This would probably smooth the curves above and also uncover slightly better solutions.
b) We have restricted the site choices to the positions of known observatories. Since most observatories are on the NH summer months (when the Moon is not as high in the Northern sky) there is a handicap.
c) The method is slow - because the altitude of Sun and Moon are evaluated from very precise routines. Simpler and faster expressions for altitude could be used - but one for each observatory would be needed.
Data reduction issuesPosted by Daddy-o Sep 28, 2012 03:25PM
post, we noted that there is a daily progression in the obs/mod ratio of the DS/BS ratio itself (let's call this thing 'the relative albedo' from now on!). We speculated as to its origin.
We have now calculated the "relative albedo" with two sets of terrestrial models - one a uniform Lambert-reflectance sphere and one a non-uniform sphere with ocean-land contrast but with a Lambert reflectance, while keeping the set of observations constant. We compare these two sets of results:
The upper panel shows the "relative albedo" for a non-uniform Earth and the lower one for a uniform Earth. The linear regression slope is printed on each line.
We note that the 'progression' for V has changed sign in going from a non-uniform Earth to a uniform earth. The magnitude of the slope on B has changed by 50 %.
We conclude from this that the 'daily progression' is due to the terrestrial modelling of surface albedo on Earth.
Or: we can state that we are seeing actual surface albedo variations during a nightly sequence of observations!
During this set of observations (night of JD 2456046) The B-band albedo of Earth decreased slightly, while the V-band albedo rose. This is the albedo of the half-Earth shown in the figures in the post here
For fun, we plot here the B and V relative albedos for tow nights with overlapping phase intervals:
The two nights are 2456075 and 2456016. For the B progressions it looks like there is a difference, at phase -96 degrees of about 0.15 in the relative albedo, and given the scatter of the points we might be able to confidently see a difference 1/10th of that, I guesstimate. That is 1.5% at a given point in phase (that is, time).
BBSO discuss '1-2% per night'.
If we take the above results at face value we seem to be able to track albedo changes through a night. Averaging sucha progression, in order to get the 'nightly mean value' would, of course, yield an average value, and an average can be affixed with a 'standard deviation of the mean' which can become very small if you have enough points - and if it gives any meaning to average the points. It does not give that much meaning to average the above progressions since they change on scales longer than our observing sequences.
But anyway, we need to start understanding what BBSO does in their data-reduction. Notably - do they assume Earth is a Lambert sphere? If so, we have perhaps arrived at their "A*" quantity? If, even more, so - we now just need to observe for a few decades!
Given only 1.5 years of data we should for now focus on how we present the results, and how we get something 'meteorological' out of the data - as opposed to 'climatological'. For instance, does the level-difference between nights 2456016 and 2456075 correspond to a change in albedo in the relevant areas of Earth that we can determine from Earth observation data from satellites? More clouds, perhaps? 2456016 is the upper sequence.
Let's see! (more to follow...)
Data reduction issuesPosted by Daddy-o Sep 27, 2012 12:49PM
For all our observations, we plot the lunar phase (in the convention we use) against the azimuth (degrees East of North) of that observations.
We see that most positive phases are observations East of the meridian; most negative phases are West. Only rarely has the Moon been followed past the meridian.
Data reduction issuesPosted by Daddy-o Sep 27, 2012 11:38AM
We consider now the DS/BS 'ratio of ratios' - that is, the DS/BS ratio in observations divided by the DS/BS ratio in the model images. We have used the Clementine albedo map scaled to the Wildey levels, and Hapke 1963 reflectances in the model images.
We look at the data from several nights in the phase-interval from -130 to -50 degrees (Full Moon is at 0 degrees; negative phases implies
we are observing the Moon in the West (setting)):
What we see are short sequences of points - each sequence covering one night. We note how many of these nightly sequences 'dip' during the progression of one night. We note the smoothness of each nightly sequnce - this bodes well for our original idea that albedo could be extracted with high precision. Jumps from night to night are more troublesome! But we need to first understand the 'progression'.
Inspection of the time-stamps on the data in a 'progression' shows that time moves to the right - that is the 'ratio' plotted starts out high at a larger negative phase and ends as a lower ratio at a less negative phase (moves left to right). Since the observations are in the West we are looking at a setting Moon, or increasing airmass along a progression.
On a single night several things happen:
1) the Moon is seen through different airmasses, and
2) the Earth turns.
[recall that the synthetic models contain all the correct distances as functions of time, so effects due to that should divide out].
1) If the effect is due to the increasing (or decreasing) air mass we need to understand how a ratio of intensities taken from two areas on the Moon, obtained from the same image, can show a dependence on airmass. Differential extinction across the lunar disc? really? We are looking at 10-30% changes in a single progression!
2) Since these observations are all at negative phase they may all represent the same configuration of Earth and Moon - i.e. we could be seeing the effect of the same parts of the Earth rotating into view through the night. From a fixed observatory there is a tendency for the same areas on Earth to reflect light to the Moon on subsequent nights - in our case either the American sector and oceans on either side, or the Austral-Asian sector.
We should not forget 1 - but let us turn to 2 for now. We can extract 'scenarios' for the Earth at the moments of observation corresponding to the data-points above. Inspectingh the observing scenarios could perhaps teach us what is going on.
We select the progression near -91 degrees. That is a set of observations from JD 2456047.75 to 2456046.87, or about 3 hours (the date is April 30 2012). At the start the Earth was in this configuration (crosses on the globe show the part illuminated by the Sun):
While this is the configuration at the end islike this:
That is, the Earth has rotated slightly during the observations and
more of the Asian landmass has swung into view, meaning that more of the
earthshine is coming from the Asian landsmass, than at the start.
Can we estimate the expected change in Earth's reflectivity during this sequence? Yes. We have quite elaborate code for that, know as "Emma's code" due to a student who worked on it in Lund. It shows the Earth from any angle at any time, with superimposed albedo maps of clouds as well as continents. A reflectance law is imposed. I think there is even some Rayleigh scattering added to give the 'bright edge' of the Earth (see any Google Earth image at startup, as you zoom in - the edge is bright, and that may even be realistic).
That code is quite complex and is not run in a jiffy. [We need more students!] But we can make a perspective model of the Earth using simple IDL routines. We can wrap surface albedo and cloud maps onto a sphere and view it from the angle the Moon is at, for any given time. There is no reflectance imposed - just the albedo maps and the perspective view. The two situations above look like this with cloud maps for the moment of observations (taken from the NCEP reanalysis product TCDC):
The simple average of the brightness of the two images above are 42.6 for the former and 40.6 for the latter - so it is getting dimmer by about 4% as time passes - perhaps because bright Australia (cloud-free at this time) is entering dusk. The cloud map images are available at 6 hourly intervals only, so there has not been time for the cloud image to change - it is simply rotated and other bits have become hidden by the day-night sector advancing.
Post-Obs scattered-light rem.Posted by Daddy-o Sep 26, 2012 04:17PM
We have extracted the mean value of alfa (the parameter that describes how broad the PSF for a given image is) for a given night, and the corresponding value of the extinction coefficient for that night. We have only few nights where the extinction could be determined.
The plot of one vs the other looks like this:
(download and look, etc)
We see that there is a tendency for alfa to be narrowly constrained, and that the extinction has a broader distribution. In general there is no strong relationship between the values, but if we ignore outliers and the effect they have on the regression (plotted as a red line) we see a general tendency for high extinctions and small alfa values to be related: For B it is quite clear. V would be clear but for the outlier, IRCUT also seems to be clear. VE1 and VE2 are all over the place. A broad PSF is given by small values of alfa. We expect broad halos (i.e. broad PSFs) on hazy or turbid nights - nights on which extinction also should be large.
Factors that determine scatter in alfa are things like image focus and how the nonlinear fitting routine determined it should stop. Physical factors include haze and thin cloudyness on the night in question.
Factors influencing the scatter in extinction include the actual regression: we used all nights with more than 3 observations used to determine the airmass vs extinction line, and for which the determinations from 3 regression methods agreed to a S.D. of less than 0.02. The three methods were - ordinary least squares, and two 'robust' methods ("ladfit" and "robust_poly_fit" in IDL). While doing the actual regression it was necessary to eliminate some outliers by hand. Physical factors include haze and cludyness and whether the halo around the Moon was well captured inside the image frame.
The relatively low value of alfa for the VE2 filter is still not understood.
Post-Obs scattered-light rem.Posted by Daddy-o Sep 20, 2012 11:41AM
At the moment we are displaying our results by providing plots of ratios of the DS/BS ratio in observations, to that in models. We do this chiefly to get rid of common factors - such as solar irradiance and distance-related geometry.
What remains in such a 'ratio of ratios' are the effects of:
1) The Earth's actual albedo,
2) the model's Earth albedo,
3) Earth's reflectance (real vs modelled),
4) lunar surface albedo's in the reference patches (real ratio vs model ratio)
5) effects due to the choice of lunar reflectance model.
We are really only interested in 1.
2 is an assumed value so that the results we get for terrestrial albedo are relative to that choice. We use a value of 0.31.
3 is a choice - we expect that any errors made in this choice will be seen as a phase-dependency in the results, and we can therefore control or at least understand it. Earth is more Lambertian than the Moon. The Earth has edge-darkening, the Moon has very little. We use a Lambertian model for an otherwise uniform Earth.
4 is observable, but only with difficulty - you need a good total lunar eclipse, then the albedos in the two patches on the Moon - or their ratio - can be measured. BBSO has done this. We have not (yet). In the model we make a choice, based on which lunar albedo map we use. As long as it is fixed the results will be relative to that choice. Perhaps we can use published images of total lunar eclipses to extract the ratio?
5 is a choice - we have models for Lambertian reflectance, as well as the Hapke 63 model and other, as yet untested, more advanced reflectance models. We expect that incompletenesses in these models will be seen as a phase-dependency in the results.
Of the above only some could possibly induce a phase dependency: 3 and 5.
We have reduced all data using both lunar models based on the Lamertian reflectance and the Hapke 63 model. We show them next - look here
for a discussion of what we are actually showing: 'ratio of ratios' and all that:
(as before, plot needs to be downloaded as it does not show up well on this blog).
The plot is for 5 filters with the EFM method applied. The left column is the ratio of DS/BS in obs to the same in model, while the right column is DS/total in obs relative to models, where 'total' means the disk-integrated brightness of all source counts (plus a few stars that we can ignore!).
The first page is for the Lambertian lunar reflectance. The last page is for the Hapke 63 reflectance.
We notice that left and right columns are quite similar. We notice that the 'jump up' at angles corresponding to about 40 degrees from New Moon is much smaller in the Hapke 63 model results. Inspection of the files that correspond to the individual points in the 'jump up' and those next to the jump, reveals that the 'jumped up' points have a different processing history: they are the result of coadding and averaging single image sequences, while the rest are stacks of observations that were averaged. Why these should be different is unclear, as yet.
We conclude that there was an effect of lunar reflectance model on our results - and that Hapke 63 is better than Lambert. This is not surprising as the Moon is well-known not to be Lambertian in its reflectance.
So - we are beginning to see observations constrain theory!
There is still some scatter to account for, and we shall return to this mattrer, using the estimates of nightly seeing available from the measured alfa.
There is also some 'slope' in the result wrt phase - so the EFM method has a success rate that is phase dependent. WHile the second-best method (BBSO logarithmic; not shown here) has some 'upturn' towards Full Moon (center of plot), the EFM has an even slope down towards FM. Perhaps some empirical fine-tuning of the method will remove this problem too.
Post-Obs scattered-light rem.Posted by Daddy-o Sep 19, 2012 09:59AM
In a previous post (here
) we have compared the Wildey and Clementine albedo maps. These maps are important for our synthetic modelling code since the albedo (along with reflectance assumptions and correct geometries) are the basis of constructing realistic model images, used in analysis. We can compare these two maps very directly by accessing both and plotting the mean albedo in boxes at common lon,lat positions:
Evidently, the Clementine albedos are lower than the Wildey ones by a factor of two for dark areas and by some tens of percent at brighter areas.
This may be an explanation for the discrepancy we have between observed morning/evening brightness ratios and modelled ones, which Chris Flynn put his finger on.
We seem to have [not shown, but material could be inserted] an observed difference (expressed in magnitudes) of 0.12-0.14 magnitudes at absolute phase 90 degrees between the morning and evening integrated brightness. In models, based on Clementine albedo and either Hapke 63 or Lambert reflectances the difference is more like 0.3 magnitudes. If the Wildey map is more correct than Clementine, in terms of the highlands/mare albedo, then using the 'flatter' Wildey albedos in the synthetic code would help on the morning/evening brightness issue by lowering that ratio in the models.
The issue with using Wildey instead of Clementine is that Clementine is a global map - Wildey only covers something like -89 to 89 degrees in longitude and something similar in latitude so there is no remedy for modelling under lunar libration. We could 'scale' the Clementine map, using the above relation and see what that gets us, though.
Before we do that, we should fully understand how our own synthetic code uses the Clementine map - I believe there is more to its use than merely being used as a lookup-table. Hans will be able to tell us about this.
A second-order robust polynomial fit to the above data (where points where Clem > Wild have been omitted) is:
Post-Obs scattered-light rem.Posted by Daddy-o Sep 18, 2012 10:34AM
Earlier we showed
the performance of the linear and logarithmic BBSO method. Now we have added the EFM method.
As before, plots are not shown well on this blog, so please download this pdf file and look at it:
You see the same 5 panels on each page - one panel per filter. On the panels you see the behaviour of the 'ratio of ratios' against phase - that is, the DS/BS ratio in observations relative to a set of synthetic lunar model images.
First page: RAW data - i.e. no scattered light has been removed. Full Moon is at the centre of the plot so we see the effect of scattered light - the obs/model ratio is increasingly not 1 as we near FM.
Second page: the linear BBSO method has been appllied to images. We see a reduction in scattered light - points 'move down' towards the 1-line.
Third page: logarithmic version of BBSO method - a slight improvement is seen.
Fourth page - the new one: the EFM method - a quite large improvement is seen over the best of the others! Many of the selected points are now lying on a flat seqeunce, except points towards New Moon, and some outliers.
The EFM data has been 'selected' in the sense that the parameter alfa, determined from images, has been used to select 'good cleanups'. A histogram of the detected alfa values was made and a notable peak found and only images with alfa in a narrow range were used for the above plot. Alfa between 1.67 and 1.73 were picked. The absence of VE2 points is due to this - the peak of the VE2 alfa distribution is between 1.60 and 1.66. We must investigate whether these are 'good' solutions. [added later: a brief visual inspection seems to imply that the 1.7 solutions are the good ones - not the majority, which for VE2, lies in the peak at 1.63. Hmm.]
Speculatively, we note the 'turnup' of points towards New Moon. We ahave earlier discussed that this may be due to some feature or failure of the synethtic lunar image model to correctly portray the Moon at large phase angles. On the other hand the 'turnups' now look less like a gradual sequence, and more like a 'jump' up. The 'jumps' occur near 40 degrees from New Moon - .... is this the 'rainbow angle'? The jump seems large - the rainbow angle paper spoke not of factors of 2 and 3 and 4 but of percentages. On the other hand the phenomenon has never been observed, as far as we know. It would be nice to have points 'outside' the rainbow angle to see the jump go down again (if this is the rainbow, that is). We note here that as we get closer to New Moon the lunar sicle is narrow and it becomes increasingly important to place the 'patch' in which the BS brightness is measured - here is an opportunity to experiment with DS/total ratios, instead.
Work continues ...
Showcase images and animationsPosted by Daddy-o Sep 14, 2012 02:35PM
We have been using the Clementine mission lunar albedo map for our work. There is an older digital map, by Wildey (The Moon, vol, 16, 1977), which we obtained access to thanks to Tom Stone of the USGS.
Here is a graphical comparison of the two:
Here they are again, now interpolated to the same 2.5x2.5 degree grid so that they can be compared numerically, along with their relative difference:Upper left: Wildey map. Upper right: Clementine map. Lower left: the relative difference = (W-C)/C. There are some interpolation artefacts on the rhs of both maps as well as in some of the edges, brought about when the full-globe result from Clementine (orbiting lunar satellite) and one-side-only result from Wildey (ground-based telescope observations) are accessed.
The map of the relative differences reveals an overall albedo offset with Wildey being darker than Clementine. In the Mare areas differences of 40-50% are found whil ethe brighter highlands have differences in the 5-20% range. The differences in albedo is therefore not a constant but depends on albedo.
We see no obvious longitude or latitude dependence in the difference. This may be relevant to whether we are interpreting the Clementine map correctly in terms of angle-dependence in the conversion from observations to the map. Or the same convention was used in the Wildey map, produced 30 years earlier!
[added later: more here
Post-Obs scattered-light rem.Posted by Daddy-o Sep 13, 2012 11:05AM
We evaluate the effect of two different scattered-light removing techniques on our data, comparing to no removal at all. We do this by considering the DS/BS ratio - and we look at the ratio of this ratio in observations to the same ratio in models - so it is a ratio of ratios, ok?
The graphics do not reproduce well on this blog so I put a link to a pdf file here: download it and read on.
There are 3 pages to look at, and on each page there are 5 panels - one panel per filter, one page per data set.
The first data set shows the ratio of ratios extracted from data where only a bias has been subtracted, plotted against lunar phase. Second page shows the ratio of ratios when the LINEAR BBSO method has been applied, and the last page shows the ratio of ratios when our logarithmic variant of the BBSO method has been applied.
Full moon is at phase 0.
We see that there is a scatter and there is a systematic dependence on lunar phase - the points 'curve up' towards full moon (the middle) and towards the edges (new moon).
Near Full Moon it becomes increasingly difficult to remove scattered light because the BS is closer and closer to the patch on the DS where photometry was extracted. The Eartshine is also weaker and weaker as you approach Full Moon because that corresponds to approaching New Earth.
Therefore a hypothesis was that the 'curve up' towards Full Moon was due to incompletely removed scattered light. Looking at the raw vs linear vs log methods it is evident that this 'curve up' is substantially reduced by application of either of the scattered-light removal methods. The linear method does well and the log method adds an increment of improvement.
I believe these plots are a demonstration that BBSO linear is not perfect - and that small improvements are possible. It is open to discussion what BBSO has done about this problem - light seems not to be completely removed in their method. How do they average their data to compensate? We should note that our image scales and observing methods (ND filters; coadd etc) are not identical.
Soon we will add a third method - the EFM!
Post-Obs scattered-light rem.Posted by Daddy-o Sep 07, 2012 09:12AM
By running the computers for one week we have been able to process all observed images (both singles and those that had been coadded from stacks) with the EFM method, and thus estimated the exponent, alfa, that the basic PSF must be raised to to model the halo around the Moon. Plotting the histogram of 9000 values for alfa we get:
We see a broad distribution from about 1.3 to 2.1. Inspection of the results reveals that only the images with alfa values very close to ~1.7 are any good. The lower values correspond to hazy or even partially cloudy nights. The values higher than ~1.7 have yet to be examined.
A typical (histogram equalized) image of the residuals for one of the images with alfa near 1.71 is here:
A 'slice' across the image shows this:Upper panel is the slice - black being the image, red being the fitted halo; second panel is a detail view of panel one showing the DS, vertical dashed lines showing the limits to the sky on which the halo is fitted; third panel shows the difference between the black and red curves in panel two.
We see that the EFM method has been able to fit the sky so that it is essentially just noise - even on the BS (right) side of the disc, and that the DS has been revealed for a wide area onto the disc itself, only near the terminator is there a problem with the subtraction - the residual dips down.
Data reduction issuesPosted by Daddy-o Sep 05, 2012 09:09AM
As discussed here
, the CCD camera became twisted in its thread on the telescope at one point. The problem was fixed, but this means that some of our images have a slight rotation about the frame mid-point. This influences the success of subseqeunt data reduction steps: especially the steps that depend on extracting flux from specific areas on the lunar surface.
We therefore tested for the presence of a rotation angle by correlating a synthetic image made for the observing moment with each and every observed image, rotating the synthetic image until the correlation was maximum - in 1 degree steps.
We plot the detected best rotation angle as a function of image sequence number and date:Top frame: detected image rotation angle vs sequence number, Bottom panel: angle vs observing day since start.
It certainly seems that almost all images up to number 1800 or so has a rotation angle of some -7 to -8 degrees. That seems to correspond to just a few nights near night 40-50. The detection of rotation is a bit spotty so there are also other episodes where a rotation angle other than 0 is detected - such as images 2000-2500. That more intermittent episode corresponds to a few nights near night 180, but there are a few more examples near night 220.
The CCD twist was correctd by Ben on JD2455991, and this datum is shown as a vertical dashed line in the plot above. Since this is not consistent with the angles measured we have to say that the test so far has been inconclusive!Added later:
Actually, it was not impossible to inspect the relationship between model images and observed images visually and to confirm when an obvious image rotation was present. Partial results (note: more points than above) look like this:
Here color coding indicates in red
the images that so far obviously have a rotation problem, and in blue
images that show no obvious problem.
The presence of blue symbols at large rotation angle must be due to failure of the algorithm for detecting rotation! It is not an easy problem to solve - at New Moon there is precious little to correlate images on, unless the DS is used - but the presence of the halo gives problems, so that histogram equalization is not an obvious remedy.
A fixed derotation for the detected nights could be implemented - this affects some of the early observing nights where single images were taken (not stacks).Added even later:
By manual inspection and image comparison, the following de-rotation angles were found for the JD in the beginning of our sequence:
Apart from 2455859 all angles before 2455886 were clear to find. 2455859 was hard to inspect as it is very near New Moon and almost no features were detectable.
As a working hypothesis, let us assume that all images before 2455886 must be rotated by something like 6 or 7 degrees, to bring them into good alignment with their synthetic models.
Post-Obs scattered-light rem.Posted by Daddy-o Sep 03, 2012 03:26PM
The results for applying the linear BBSO scattered-light removal method to all data are further considered. Here we show data where the lunar disk is well centred and the radii extracted are between 131 and 149 (real range is 132-148) pixels. (Sorry about the image rotation!):Rotate image and then use this caption: Results from comparing model images to observed images - models calculated for the observing moment but no halo generated. In the observations the halo has been attempted removed via the BBSO linear method. Column (1) DS counts to total counts for observation (black symbols) and same for Model in red symbols, against lunar phase; Column (2) observed DS to total ratio against airmass; (3) ratio of the DS/tot ratio for observation and models.Note that Full Moon is at 0 degrees phase.
We note that for lunar phases between New Moon and about 100 or 110 degrees the model and observed ratio of DS to total flux behave similarly. For phases closer to Full Moon the model DS/tot ratio is much smaller.
That is probably because near Full Moon the halo gets closer to the DS patch where the DS is measured: In the model no halo is present so the DS brightness is not polluted until the patch is actually part of the BS - in the observations the halo moves with the lunar terminator towards the BS and becomes harder and harder to remove so that it starts to pollute the DS - even if measured in images with some of the halo removed.
In the middle column we see a fair spread in observed DS/tot vs airmass. Most of the observations are for AM < 3, while a few have even higher airmasses. In the bulk of the observations we see no filter-to-filter consistent evolution of DS/tot ratio against airmass, and conclude that there appears to be no dependence on airmass; this is good as the whole point of simultaneous observing of DS and BS is that external factors, like extinction, affect both sides identically.
In the last column we see the ratio of the DS/tot number between observations and models. The behaviour noted from the first column, above, is evident here - the DS/tot ratio in models is much smaller than in observations for phases inside 110 degrees or so and the ratio of ratios goes up accordingly. For phases between 110 or 100 degrees and New Moon we see a more constant behaviour consistent with the observation made above that model and observations behave similarly. We see, however, a consistent pattern from filter to filter - there is some 'curvature' with phase. The inner part of this could be due to lingering halo effects, but the outer 'curve up' must be due to effects in the model, not in the observations. Near New Moon it seems that the DS/tot ratio for observations grows compared to the same ratio for models - the models' DS is not as bright as in the observations near New Moon. We note that the DS model brightness is a function both of the reflectance description of the Moon and of the Earth.
1) There is indication that halo-removal near Full Moon (phases between 0 and +/- 100 degrees) is increasingly incomplete.
2) There seems to be little or no effect of airmass on the DS/tot data.
3) There is some indication of a reflectance problem in the models for phases approaching New Moon - either in the Earth description or in the Moon.
1) is good news - we thought we could only do removal up toi half Moon - the limit seem sto be 10 or 20 degrees beyond that, towards a fuller Moon. 2) is very good news - it opens up for our method to be applied at low altitudes, i.e. for small lunar phases near New Moon.
3 should be investigated by now redoing the whole reduction but with some simple and single change in the modelling - we cannot change the BRDF for Earth (it is always Lambertian), but we can change the lunar BRDF from Hapke 63 to Lambertian.
In these models the Earth was modelled as a 'cloud free Lambert sphere' with ocean/land contrast built into the single-scattering albedo. The Moon was modelled by the Clementine single-scattering albedo map with Hapke 1963 reflectance.
Data reduction issuesPosted by Daddy-o Aug 31, 2012 02:00PM
Here are some of the first results from applying the BBSO linear method - the reductions are slow so more will be on hand later! Sorry about the low quality in the image - there must be a way to do it better, but ...
Rows 1-5 give results for each filter. Ignore columns 1,2 and 3 for now - they are diagnostic. Column 4 shows the ratio of observed terrestrial albedo to modelled terrestrial albedo as a function of lunar phase.
We pick out the DS as either Grimaldi or Crisium depending on which is in the DS. We then calculate the observed DS brightness divided by the brightness of the whole disk. We then do the same for the synthetic model, and plot the ratio of the two.
This may seem a strange quantity to plot, but consider that in the (unlickely case) that we both had perfect observations and the model was correct in all aspects, then we would see a ratio of 1.0.
If the model is somehow wrong - for instance if the phase function it is based on is unrealistic then the ratio would have a phase dependence.
As it is, we do not have perfect observations and we see a fair bit of scatter. The scatter comes about for several reasons - first of all the observations have Poisson noise - we are extracting a small 4x4 pixel patch on what is the dark side where counts are probably on the average of 5 or so. Additionally we have noise from the alignment between the actual Moon and the coordinates we have calculated from which to extract information - there is a missmatch of up to several pixels here, so for a small area like Grimaldi a few pixels error in placement brings you into the bright surrounding areas. For Crisium, which is larger, this is less of a problem.
Finally there is a still un-solved problem with synthetic models and observations apparently being off by some small amount in terms of a small rotation. This may be from the days when the CCD camera was actually physically twisted by a few degrees in its placement on the telescope. In such cases the intensities of the pixels extracted in the synethtic model and observed image are even more different.
So, some things to work on are:
1) Use larger DS patches, so that the Poisson statistics are not as much of an issue.
2) Put the patches in uniform areas on the Moon so that missalignments do not cause acquisition of contrasting areas. Inside large, even Mares or on the brighter highlands.
3) Use better estimates of disc centre and radius.
4) Figure out a way of aligning the synthetic model and the observed image better.
PS: More data are available all the time so the figure above will update now and then.
Have implemented 1 and 2, by enlarging the area inside Crisium that is used, and using a rectangle in the highlands south of Grimaldi instead of Grimaldi itself. Also trying 3.
Post-Obs scattered-light rem.Posted by Daddy-o Aug 30, 2012 12:50PM
The linear BBSO (i.e. as done by BBSO) method and the modification that operates on logarithmed images can be compared:A strip, 20 rows wide, across the image was made and the rows of that strip averaged, and the average plotted. Black is the original bias-subtracted image, red is theordinary linear BBSO method and blue is the log-method. We plot the absolute value of the fluxes to avoid problems with log.
Both the linear and the log method does well on the sky at left, but there are differences on the edge-near disk. Tests on synthetic images, where we know the flux to expect on the disk, has shown that the log-method is more accurate than the linear method.
The residual mean (about 2) is unsettling, but has to do with the use of absolute values and the averaging over several rows - inspection of the images reveal values distributed around 0.
Data reduction issuesPosted by Daddy-o Aug 29, 2012 03:16PM
We have to determine the radius and centre of the lunar disc in order to reduce observations.
In doing that we must be aware that differential refraction causes the Moon to appear non-circular as it comes closer to the horizon. Using a formula from the Nautical Almanac Explanatory Supplement we generate the following table for 600 mmHg, 10 degrees C and 30% relative humidity:
Z d_refr am
27 1" 1.1
69 6" 2.8
75 12" 3.9
where z is the zenith distance in degrees, d_refr is the differential refraction in arc seconds and am is the airmass. The differential refraction is calculated over a 1-degree distance centred on the given zenith distance. Our FOV is about 1 degree wide and one pixel covers about 7".
We thus see that the Moon is differentially refracted by less than one pixel up to about 2.6 airmasses. Two pixels are reached about 26 degrees (4 airmasses) above the horizon.
Some of our observations are certainly close to the horizon, as we have tried to observe when the lunar phase is near or less than 30 degrees (at Newish Moon).
At 30 degrees the uncertainty in our determination of disc radius and disc centre (based on a circular assumption) is thus starting to be challenged by the differential refraction.
Data reduction issuesPosted by Daddy-o Aug 29, 2012 02:14PM
In order to extract fluxes from particular areas of the Moon we need to take lunar libration into account. Based on the synthetic model code Hans wrote we can do that now.
On the left is the synthetic model. On the right is the observed image. On it are some dots - they are supposed to be inside Mare Crisium and Mare Nectaris and two other locations on the darkened side. We see that we nail Nectaris and Crisium. Doing this for every observed image we will be able to extract feature fluxes and compile DS/BS statistics.
Data reduction issuesPosted by Daddy-o Aug 27, 2012 04:22PM
Since the setting of the colour filter (as well as the shutter) was unreliable we must find a way to detect which images are taken through which filters.
Here we plot the raw fluxes (counts divided by nominal exposure time):Total lunar Fluxes plotted as magnitudes against the lunar phase (New Moon is at 0). Bright is up, faint is down.
The data for each image (black symbols), here expressed in magnitudes, are overplotted with a phase law (red) inspired by that in Allen "Astrophysical Quantities", except we modify the coefficients in that and use instead:
Notably the coefficient on the linear term is about half of what Allen specifies.
Particularly in VE1 we note the presence of two sequences of data. We have 'fitted' (by eye) the sequence that is represented on both sides of the new moon to the Allen phase law. The orphan sequence is below the adopted data suggesting that a filter with less transmission was obtained here when VE1 was requested.
For B it seems that intermittency causes some fluxes to be higher, but they do not fall in a delineated sequence so cannot be identified with a filter.
The V filter seems to have the same problem, although less so.
VE2 is also somewhat 'broad' in its distribution.
IRCUT shows two sequences.
The VE1 and IRCUT filters are extremely similar in transmission properties and are quite similar in the plot above, including the presence of the 'second sequence'.
The 'second sequence' is quite similar in flux to the V observations, and it is consistent to say that when IRCUT and VE1 failed to be chosen V was obtained instead.
This seems to also apply to some of the outliers in VE2 and B.
We thus suggest that a working hypothesis for the failure of the filter wheel is that when it failed the V filter was selected instead.
We next proceed to eliminate the outliers in each filter so that a dataset can be defined which will allow identification of the extinction laws in each filter. With that in hand it may be possible to 'tighten up' the data and move towards a 'golden dataset' from which also DS fluxes are worth extracting.
The presence mainly on the left side of the diagrams of a 'second sequence' of data implies that something like a mechanical problem is behind the FW failure - because the telescope is flipped over the meridian to observe mainly in the East or the West depending on whether the Moon is rising or setting (before or after New Moon).
Post-Obs scattered-light rem.Posted by Daddy-o Aug 27, 2012 11:36AM
In order to precisely remove scattered light from the observed images we need to know the centre of the lunar disc in image coordinates, as well as the radius. These numbers are used by the BBSO method, while the EFM method can work without them. Extraction of fluxes from designated areas of the Moon also requires knowledge of the disc coordinates.
We have, as described elsewhere, found a fairly good way to estimate disc centre and radius, and have more than 5000 images (singles or sums of stacks) with know coordinate estimates.
We can check on the quality of these data by inspecting the time evolution of the disc radius in terms of the lunar synodic period (27.322 days).
Plotting the detected radius against observing time modolu 27.322 we get, for the 5 filters:
There seem to be some outlier groups, as well as a general scatter. The scatter is on the order of 2-3 pixels while the outliers reach 5. These outliers can be identified and the relevant images inspected.
We fit a general sine curve to the data, and get:
Offset Amplitude Period
141.169 7.98152 27.6367
+/- 0.0108026 0.0166443 0.00208776
140.663 7.81091 27.5796
0.0122441 0.0199136 0.00187041
140.647 -7.47896 27.5736
0.0413606 0.0837630 0.0111588
139.845 8.51341 27.5771
0.0245651 0.0383531 0.00435187
140.744 7.98575 27.6255
0.0358718 0.0596396 0.00708377
The period is not close to the expected 27.322 days. We expect this is due to a poor fit (in turn due to the outliers). We identify the outliers. 108 images are found that have radius more than +/-3 from the fitted sine curve.
Upon inspection, it turns out that not many of the identified outliers are obvious 'bad images'. The determination of radius and disc centre is therefore somewhat deficient.
Real World ProblemsPosted by Daddy-o Aug 13, 2012 03:06PM
The image below is taken at the IIWI computer with the CCD camera
attached to the board, sitting in a PCI slot on the IIWI. The software
used is SOLIS.
This image looks like things we have seen before. The broad stripes are reversed but that may be something in the display software settings. It is a very primitive setup - all I have managed to do is take a
picture at some long exposure time. The CCD is still attached to the
telescope, I think, so the shutters are closed and what we see is a flat+bias
frame. As it was dark in the dome when the image was taken the 'signal' is dark current - not light. The bands are structures in the flat
field. The spots are noise. The large spots are possibly CR hits?
I think the image proves that the camera is able to take pictures. The
noise may be due to a damaged cooler or - more probably - that the
cooler is not switched on (I don't know how to do that) yet.
It is at least an image from the camera - so board and camera must be
OKish. Why then are there no images when the camera is attached to the PXI? One
answer could be that the PXI was damaged during the MLO power surge. I
am leaning towards that theory now.
Added later: Here is an image with cooler ON.
The minimum values is 402 - a bit high, but at least it now looks like a bias frame. I would say that there is nothing wrong with the camera or its board. So the problem must be in the PXI!
Real World ProblemsPosted by Daddy-o Jul 10, 2012 10:50AM
July 10 2012: A power problem at MLO has been reported. At the moment the power is back, and all our machines - except
the PXI can be reached.
If the PXI is damaged this could be the end of current efforts to use the telescope.
Hopefully access will be back soon and we can gather some more data, before the decisions we have to make in September.
Real World ProblemsPosted by Daddy-o Jul 09, 2012 03:23PM
Following on from the Data Summary post
, we have reviewed all the good data and built an understanding of what the flux-ratios are between B and the other filters. We then use that ratio to test all observations to see if the FW was behaving as expected or showed signs of malfunction. The malfunctions often mean that all fluxes are the same, consistent with one single filter being set instead of the sequence of filters the script asks for.
Focusing then on only those nights where the FW was well behaved we extracted all total fluxes (total bias-corrected image counts divided by nominal exposure time), limited the data to airmasses less than 4, removed the lunar eclipse, and then plotted all fluxes against lunar phase:(sorry about the strange cropping!) In different colors we see the extinction-corrected fluxes as a function of lunar phase. We only show data between 30 and 90 degrees (90=half Moon) since this is all we can acquire in co-add mode. The blue,green,red and orange symbols correspond to B,V,VE2, and VE1+IRCUT (very similar filters), respectively. The curves are 4th order polynomia fitted with a robust code that omits outliers.
We see some scatter around the lines - some of it (e.g. near 50-55 degrees is probably clouds. Some of it, in IRCUT/VE1 near 90 degrees may be 'no filter was inserted at all', or shutter exposure time was longer than requested.
From the fits we can generate flux-tables for use in identifying the remaining data: There are many more images available but they seem to be with unknown filters because the filter acquired and requested were simply not the same. These images may still be of use in DS/BS and albedo analysis - we just need to figure out what filter was used!
Apart from causing many lost images (i.e. we get a bias frame instead) the shutter semes to mostly work as expected when it opens at all. The Filter Wheel, however, selects random filters (as far as we presently know), or no filter at all, when it does not work.
The present work enables a data-selection filter for use in post-observational processing.
Note that the present data are not relevant for DS/BS studies as scattered light has not yet been removed. First we have to identify the images that can be further analysed!
Mechanical designPosted by Daddy-o Jul 03, 2012 09:37AM
The above is the result of testing flux constancy in images taken of the Hohlraum lamp on a given night, through all filters. Since the lamp is constant and the telescope is not moved then, provided that the FW and shutter works, it should be possible to get constant fluxes for each filter. The above shows that this is hardly the case.
In the first five panels (left to right, top to bottom) we see histograms of the fluxes (cts/s) derived by opening images, subtracting the bias, and extracting the total flux as well as the exposure time and filter name from the headers. We see that in no filter is there a prevalent flux - that is, either the lamp altered its brightness or the FW never acquired the requested filter or the shutter was miles off (so that the requested exposure time was nowhere near the one we got). The bottom right plot shows the fluxes plotted in order of acquisition - since we see straight sequences I think the lamp is not the problem - but the shutter and or the FW is.
Near the top we see a special pattern - this is understood when inspecting the list of of filters and fluxes:
We see that when the filter supposedly changes, the flux is all strange given the otherwise regular sequence.
So: never use the last or first image in a sequence from a stack!
Whether this has been the case throughout the 1.5 years of data we soon have, is to be revealed by further analysis. For now, let us simply reject all first and last images in all sequences.
A quick look at the NGC6633 images shows the ABSENCE of the above problem:
The filters did NOT act strange when changed. So - more intermittency, but also a lesson: check all sequences of images for the above problem!