Showcase images and animationsPosted by Peter Thejll Oct 10, 2015 10:21AM The recent total lunar eclipse gave an opportunity to apply photometric methods. It is possible that the light from the Sun, passing through the Earth's atmosphere is coloured by processes that ultimately also have a climate effect on Earth. Recent volcanic eruptions may cause the eclipse to appear more red than usual.
Can this be measured accurately?
The Internet was overflowing with images and image sequences of the eclipse. Among them this composite image by K. Lewis (www.photosbykev.com):
Using the method described in the paper by Park et al, we can convert the RGB values from the image file above to Johnson B and V colours (the possibilities are limited to these two bands - more can be done with the NIR filter taken out of the camera).
We extracted the mean and median B-V colour from the sub-frames above, and plot these values in a sequence, starting from upper left and ending in lower right frame:
We see the reddening of the Moon as it is eclipsed. The dotted horizontal line is a nominal value for B-V=0.92 based on literature values for the Full Moon, and the measured values from the composite photo were adjusted up by about 0.6 in B-V to roughly coincide with the 0.92 level. We see that the eclipse was redder than the Full Moon by about 2.5 in B-V.
The jagged outline of the curve, and the evident jumps in brightness in the main photo, seem to indicate that the detector used was not linear - this could be because the images were obtained not only on the linear part of the CMOS devices' transfer curve.
We obtained other images of the 2015 eclipse and also converted the RGB values to B-V, but found eclipse B-V very different from the above. Attention to obtaining Full Moon images just before and after the eclipse takes place is required, as is some attention for exposure levels, trying always to capture the various stages of the eclipse at mid-sensitivity range of the device used.
Also important is that there are gradients in the shadow of the Earth on the Moon and the geometry of the eclipse and the times of observation - and the location on the lunar surface used - all influence the results obtained.
Showcase images and animationsPosted by Peter Thejll Aug 16, 2015 10:29AM This is a trial video of all our good V-band images. On the left is the linear image, on the right a somewhat histogram equalized image. Want to embedd this video, but here is a link until further notice.
From flux to AlbedoPosted by Peter Thejll Aug 07, 2015 11:34AM NASA recently released a set of images showing the Moon crossing the disc of Earth, as seen from the DSCOVR satellite orbiting Earth in the L1 Lagrangian point. From that point the satellite can keep constant watch on Earth, with the Earth disc always fully lit. NASA image, from the DSCOVR satellite.
This image is interesting because it brings together so many different pieces of information, and possibilities. With the Moon an unchanging object (surface cratering might change on timescales of 100s of millions of years) in the same frame as Earth it becomes possible to very accurately calibrate photometry of the Earth.
Why do we need good photometry of the Earth? Well, weather satellites already give us a flood of images of Earth but the images - especially from older satellites - are mainly suited for weather forecasting - "Here is a cloud mass, and it is moving this way, so we think it will rain or be cloudy at such and such a place a little later". This sort of information is of enormous value to society on a day-to-day planning basis, but for climate research the data have to be absolutely calibrated - in other words, it is no good if the pretty pictures of Earth are taken with different instruments that have different sensitivities, or with instruments that slowly alter their properties. The resulting data would not be inter-comparable.
Climate research needs to have data for Earth that tells the story of what happens over time - this is the difference between 'climate' and 'weather': Weather is tomorrows' events, but climate is the mean of what things were like 20, 50 or 1000 years ago. If changes in climate can be detected, then we can start thinking about what causes the change. A lot is known already, but every bit of new, reliable information can be put to good use.
Looking at the DSCOVR images we note the cloud masses on Earth - if you have seen the animated film showing the Moon passing across the disc you will realize that the Earth is turning as we watch - so new cloud masses are being brought into view while others pass beyond the limb of the Earth causing variability in the average brightness of Earth. Link to the animation is here.
Using image-analysis software it is possible to pick out the parts of the image that are Earth and the parts that are Moon, and look at the statistics of the numbers. The images are 'RGB' images and each RGB-image contains one image for the red light, one for the green light and one for the blue light. Because the Moon is in the image there is the huge benefit that even if the camera settings or properties varied slightly in time then the ratio between Moon and Earth brightnesses in the three colour bands would be independent of these errors or slow changes.
So, with these wonderful properties of the NASA image, how useful is it, in reality? The amount of clouds on Earth relate directly to the 'radiative balance' that drives climate. More clouds mean less light get into the climate system to warm it up. Therefore, variations in amount of clouds will alter the temperature. We all know it is cooler when a cloud draws in front of the Sun, but over larger scales and longer times, how much will mean temperature change for a given change in cloud cover?
Estimates indicate that a 1% change in global mean cloud cover will change global mean temperature by 1 degree C - roughly (could be half that, could be twice that), so if we could be sure to measure mean cloud cover to an accuracy of 1/10th of a percent we could expect to have data that gave us important information in the clouds-vs-temperature question.
In the NASA image the variations in the clouds seen, due to rotation, amount to some 3% in the short segment of images provided. The images are taken about 15 minutes apart from the DSCOVR satellite so in that short film (containing 12 images with a full Moon disc in view) we cover almost a quarter turn of the Earth. Aha, so we now realize that the amount of clouds on Earth varies by several percent from hemisphere to hemisphere and probably day to day. Next day the amount of variability is some other number and some cloud masses have dissolved while new ones have been formed - so on a daily basis cloud variability on Earth is high. Uncertainty is therefore high, as to the actual number of clouds present - weather forecasting may care where the clouds are and how many there are, but climate investigations need more precise (less scattered) data.
If we average many images of Earth the variability would average out and the uncertainty go down. How many pictures of Earth, like the ones here, would you need to have for the average cloudiness to be of use in the clouds-vs-temperature question? The answer turns out to be that with about 1000 independent images the average would have an uncertainty of 0.1%.
How do we get 'independent' images? The phrase refers to 'statistical independence' and has to do with how long you must wait for a fresh cloud-scape to be present over the visible disc of the Earth. Weather systems can last several days so you can perhaps get an independent image of Earth every 2 or 3 days.
It therefore looks like data gathered over a decade could produce data for cloudiness that could be used for investigations into climate change mechanisms via the cloud-mechanism. This answer holds equally well for ordinary satellite data except these data have the added problem of instrument drift.
Of course, the Moon only crosses the Earth's disc as seen from the DSCOVR satellite every so often - you certainly do not get a sequence of images like the above every day - or Month. So is it useless to use these data for the climate-change purpose then?
Well, all satellites have a basic problem - they do not last long in deep space due to radiation damage from cosmic ray particles and outbursts from the Sun, so they get replaced every several years or so. This will probably also have to be done with the DSCOVR satellite, if there is funding for it. The point I make here is that the presence of the Full Moon in the image frame every so often allows a re-calibration of the DSCOVR satellite instruments that give unprecedented accuracy. Geostationary satellites are between the Moon and Earth and do not capture the Moon in-frame very often, and when they do it is as a small object peeking out behind the Earth in a corner of the picture. These chance-occurrences are, however, used nowadays to calibrate satellite instruments, by EUMETSAT and others. Classically, satellites were calibrated on bright white surfaces on Earth, such as salt deserts - this gives about 1% precision and accuracy, while the lunar methods are able to reach a few tenths of a percent accuracy.
Another comment that can be made based on the NASA image is that the Moon appears dark against the Earth. In principle this is well understood because the lunar soil is dark while Earth is full of bright clouds, icecaps and a scattering atmosphere. Desert are also very bright, but forests are dark, as is the cloud-free ocean. With the images at hand we can readily measure how bright the Moon is relative to the Earth in each of the R, G and B colour-bands. The ratio should give the ratio of albedos (reflectivities) of Moon and Earth, because the geometry is the same - Sun is behind the DSCOVR satellite and both Earth and Moon are 'Full' at the time the images were taken. I did this and find a ratio about 2 or 3 times higher than the standard ratio expected. The Moons' albedo is in the range 0.05 to 0.10 while Earth is near 0.3 so the ratio should be 1/6 to 1/3, roughly, but actually comes out as 0.8.
Why is this? I think there may be two reasons - first of all the back, usually hidden, side of the Moon is brighter than the front because there are more mountains and they are bright. The front has lunar Mare and they are darker than the mountains. Second, we are seeing the Moon under Full Moon conditions here and albedo, or reflectivity, depends on the angle of viewing, just as any surface you look along. Near vertical incidence the lunar soil appears brighter than at other phases. This is known as the 'opposition surge' and explains why the Full Moon, as seen also from Earth, appears brighter than at other times - not just because a larger fraction of the Moon is illuminated then but because a lit region simply reflects more of the incident light when the incidence and viewing angle is 0 degrees - straight down and straight back out again.
Why does albedo depend on the viewing angle? Any rough surface has shadows behind boulders and mountains and only when the angle of incidence is the same as the angle of reflection will the shadows be hidden from the observer - looking at a lit and rough surface from another angle and the shadows can be seen, leading to a lowered average reflectivity. The lunar also has small beads of glass and crystalline material which contribute to the enhanced reflectivity during Full Moon. Is there no 'opposition surge' for Earth? After all the geometry for Earth and Moon is the same here so a surge for the Moon might also cause brightening of Earth? I don't know yet - but note the pretty 'sunglint' seen in the centre of Earth during the animation.
The images used here are not 'science grade' but have been optimized for viewing online so transformation of colours have been applied in order to yield a pleasing and realistic image - perhaps the scaling has caused an offset in the brightness ratios, explaining the large Moon/Earth albedo ratio. The raw science data will be released by NASA later, and consists of 10 very narrow band-pass filters.
We might pause to wonder why all this effort is worth its while? The answer is simple and has to do with reproducibility of observations - we need to have many estimates of all properties of the climate system in order to understand it fully, and winnow out the erroneous data - a point made brilliantly by Ruth Mottram in her excellent blog sternaparadisaea.net.
As they say on Mythbusters "If it isn't reproducible, it ain't Science!"
A very clever alternative to watching Earth from Space in order to determine its reflectivity, is to stay on Earth and watch Earth from there - by using Earthshine observations! And the rest of this blog is about our attempts to do just that.
Data reduction issuesPosted by Peter Thejll Jun 05, 2015 12:10PM We have generated 'super flatfields' by considering all the flat fields we have - we have many exposures through each filter taken on sky, the dome inside and the hohlraum lamp. While the patterns are the same for the three types (mainly a roof-tile like diagonal pattern' we have dips in the corners of especially IRCUT and VE2 (and VE1). It is therefore not obvious which flat field is the 'correct' one. Assuming that the ones with the least vertical relief must be the ones closest to reality we have adopted one superflat for each filter.
We need to investigate if there are other ways of testing which FF is best - and we have the possibility of 'scamp' and 'sextractor' to generate background fields, which we assume are some sort of smoothed residual after sources are removed. Using frames from NGC6633 we generate these background fields from 'raw' frames and 'bias-subtracted and flat-divided' cluster images.
A sample of the background fields, shown as contour plots, for one science frame is shown here:
The upper field is for 'raw' and the lower for 'bias-subtracted and flat-divided'. The contours seem to have a mean near 18 for both - the contours evidently are not the same. The differences between the two backgrounds amount to 0.1 - 0.2 counts.
We should center the backgrounds on sky-coordinates and average and see if the fields are consistent given method (we assume so). TBD.
If the patterns are stable we have learned that applying bias-subtraction and flat-division can alter the levels by a few tenths of a count. This may be important for photometry of the DS. Presumably it is better to correct for the flatfield than not to, but that depends on whether the flat-processed fields are consistent. Perhaps warp can be used to generate a stacked field, and its background?
Added later: Doing just that we get the average background frame for 'raw' and 'bias-subtracted, flat-divided' processing:
The relief in that field (took the most-overlapping 512x512 region) is from +0.1 to -0.02 -- so some averaging has occurred, but there is still a one-tenth of a count effect of applying the flat (in, B, and when 14 images could be stacked).
So, we cannot rule out that applying flats have a large effect compared to DS photometry - i.e. 0.1 out of 1 is 10% and at best 0.1 out of 10 is 1%.
The effect is regional inside the frame and so we can well have an effect on the limited-region fittings we perform on the DS.
Might want to consider what the effect on the BS-halo, evaluated at the DS, is if the BS also has flatting-effects brought to it.
Basically, applying flats are necessary as long as they do not bias results due to bad production of the flat-field itself.
Data reduction issuesPosted by Peter Thejll Apr 27, 2015 10:41AM Here is an example of a two-way smear. The red area is the overexposed region, and it has bloomed along vertical columns in the image. To the right of this bloom is a smear extending hundreds of pixels. What is that?
Read-out direction is top-down.
What is causing this smear to the right? Is it related to the 'depression of the sky in a broad band left-right' that we see in some images?
The intensity of the smear is at the 500 count level - i.e. it acts just like part of the halo from the BS. This has, surely some consequences for data reductions? Do images that have not bloomed have this halo? Is this why the bright edge of the BS-halo is always hard to model?
Can we see if the fitting of the BS halo is easier in images that have been modestly exposed, compared to those exposed right up to 50.000 or 55.000 counts?
Usually photometry is performed on images of sources by collecting the flux from an area near the source and relating it to the airmass of that particular observing moment. In a Langley plot the relationship between magnitudes and airmass will be an (almost) straight line, and the extinction coefficient is the slope of that line.
As light passes through more and more atmosphere there will be more absorbtion as well as more scattering. Both remove intensity from the direct beam. Absorbtion just takes the light away (at least at optical wavelengths) while scattering redistributes the light near the source. If the method of measuring light from the source therefore allows re-capture of the light scattered - for instance by allowing for large collection areas in the image - then the effect of scattering will be reduced and a smaller extinction coefficient should resultr, as compared to the extinction calculated when scattering is allowed to remnove light from the calculation.
Thus, when considering an extended source like the Moon we have an opportunity to test the magnitude of scattering vs absorbtion by explicitly collecting all light near the Moon (for instance, by summing the flux in the whole image frame) and comparing to what we find when light is collected only from a small patch on the lunar bright side.
We have done this for 5 wavelength bands and a score of observing nights where conditions were right. We measured flux in each image in two ways - either as the total flux in the image frame or as the flux inside a small patch on the lunar brightside defined by selenographic longitude and latitude. Langley plots were constructed and slopes measured using regression, along with error estimates of the slopes.
Results are plotted here:
Langley extinction for whole-image fluxes plotted against extinctions determined from a small bright-side patch in each image.
If the image is rotated and inspected (sorry about that) we see that for the five filters extinction determinations often agree, despite the expectation that extinction determined from the small patch (allowin scattered light to escape) would be larger than the extinction determined from whole image. Only a few points deviate significantly from the diagonal line.
This implies that scattering at MLO was very low, compared to absorbtion - as scattering includes Rayleigh scattering thge plot for B should be considered carefully.
In a few cases the plots allows us to identify 'bad nights' when the extinction determinations do not follow the diagonal.
Showcase images and animationsPosted by Peter Thejll Dec 11, 2014 09:45AM From the International Space Station (ISS) the Moon can of course be seen, and has been photographed with hand-held cameras. Someone took an image of the Moon with exposure settings chosen to highlight the earthshine. Below, such an image is discussed:
On the plot we clearly see the dark sky above the Moon, the entry onto the lunar disk, the extensive dark side, the bright side and the transit of the bright atmospheric layer and then entry onto the dark Earth below.
We note that the bright side is saturated as the intensity reaches 255.
We do not know whether astronauts on the ISS have available high-dynamic range cameras. We think the above image is snapped with a standard 8-bit camera.
Using 16-bit handheld CCD cameras - such as a Hasselblad fitted with a 16-bit digital back - images of the earthshine could be obtained that showed bright side and dark side at the same time without bright side saturation allowing the DS/BS ratio to be calculated. That ratio is proportional to terrestrial albedo.
Such images could be of educational value and could highlight the role that earthshine studies can have in climate change research.
Since the atmospheric profile is quite uniform horizontally in the above image, we estimate the mean atmospheric profile and subtract it from the slice across the Moon, thus revealing the profile across the Moon, without atmospheric effects:
The top panel shows, in black, the profile across Moon plus atmosphere; the red graph shows the profile of the atmosphere adjacent to the Moon; the bottom panel shows the difference.
We note that the atmosphere-free profile in the bottom panel above shows almost no 'halo' from the bright side - unlike our own images, taken from MLO, through the atmosphere.
That piece of information is quite interesting: As we understand it, our own images have a contribution to a halo from the optics of our telescope along with a variable halo due to the atmosphere scattering. From the ISS a camera, with a lens, was also used and one wonders why there is no sign of this optics-halo? Is the ISS camera optics vastly better than ours? This is not likely, since, probably, any old hand-held camera was used. On the other hand, modern lens-coatings (such as in the Canon 'Dragon-eye coating' lenses) have high standards and are designed to suppress lens-scattering and internal multiple reflections.
So, we wonder if the ISS has some special lenses at hand?
We note that the above image is for an extreme 'almost new Moon' situation with little flux from the bright side to scatter and interfere - we should see if more ISS images of the Moon, showing earthshine, at phases more like 1/4 are available? If those images also do not have any appreciable halo due to optics we might be on to something.
Note that the image of the Moon is probably taken through a clear spherical dome or window mounted on the side of the ISS - what are the implications of this for observing the Moon through a dome, here on Earth? Is distance between dome surface and camera lens a factor?
Notes: 1) Extinction not accounted for - this may cause the 'slope to the left' of the DS seen above. 2) The image was taken in August 2011 using a Nikon 3DS camera. That is a 14-bit camera taking .NEF frames. We have this frame from NASA now.
We investigate this using one of the 'good' stacks of 100 images for the V band. We call the stack 'good' because it seems that repeated fits to it with slightly different settings of the fitting algorithm give similar results.
We split the stack into 33 3-image averages, 16 6-image averages, 8 12-image averages, 4 25-image averages, and 2 50-image averages. We also fit the 100-image average several times. We fit each of these multi-image averages and plot the results: - so the conclusion is that flux constancy varies. Top left: log-log plot of how the RMSE depends on number of frames in each average. the RMSE is calculated in the areas used for the fitting of the model to the observed image. The fitted line has slope minus a third. Top right: log-log plot of how the estimated uncertainty of the Albedo-fit (given my routine LMFIT) depends on number of frames averaged. The fitted line has slope minus a third. Middle left: log-log plot of how RMSE and Albedo uncertainty depend on each other. The line has slope near 0.9. Middle right: Fitted albedo against number of frames. The error bars are the formal estimates of the albedo fit uncertainty (given by LMFIT). Bottom left: How the standard deviation of the Albedo determinations (in middle right plot) depends on number of frames averaged.To avoid 'overplotting' we have offset N slightly by adding small random numbers but still maintain a clusteringa round the actual N value.
We see some interesting things in this evaluation of 'intra-stack variability': The uncertainty (both RMSE and Albedo uncertainty) ought to drop as one over the square root of number of frames averaged, if the noise causing the scatter is normally distributed and independent and there were no biases. As it is, the errors drop with a minus 1/3 slope indicating that something is holding back error-averaging. We also see some hint in this in the middle right panel: albedo seems to depend on how many frames we average - the frames were also averaged sequentially (3-averages formed from frames 1,2,3 then 4,5,6 and so on). We can test if the order of averaging is important by doing some bootstrapping and randomizing (TBD) but there is also the possibility that increased averaging allows some subtle biasing factor to emerge. Finally, we see that intrinsic error (i.e. that not due to choice of image stack but due to factors like noise within a given stack) can be brought below 0.2% when averaging just 20 frames. We also see that the formal estimates of albedo uncertainty are not that far from the scatter of the albedo determinations themselves (but might be off by a factor of 2 or so).
So, in summary, we have found that scatter can be brought down to the few-tenths of a percent level by averaging and using our fitting method and observed images. [Wahey! That is actually a major point for the Big Paper! Rejoice!] We find that there is some sign of bias causing a monotonic evolution of the fitted albedo with increased averaging, as well as a decrease in scatter that does not drop as expected for averaged independent noise.
We can test if the order of averaging is important by bootstrapping, and this will be done. It could help rule out effects due to some time-dependence of the properties of the images in the stack.
We have now tested some more ideas: We test effect of alignment as well as effect of the order in which the images are selected for averaging inside the 100-image stack.
Upper left: red line and black points show the old result - sequentially selected frames are averaged, without alignment. Green points and line - the images are selected non-sequentially as in 'bootstrap with replacement' and are also not aligned. We see that the slopes of the lines are the same - about -1/3 but that the bootstrapped results have about 25% higher error than the sequentially selected.
This gave the idea that 'something changes' along the stack which results in increased error. It could be that the Moon drifts slowly across the sky so that mis-alignment is more evident in sums of frames selected far apart as opposed to in sequence. There could also be 'other time dependent' things going on - such as various CCD-related effects.
Upper right: to test the drift hypothesis above we next aligned the selected frames when bootstrapping. As we see there is no effect of doing this! So drift is not the problem - or is it? Alignment consists of sub-pixel shifts that may induce variance-changes due to non-conservative interpolation in the image frame. We have considered this in this blog (see discussion and further blog references HERE). Since the green data look the same in upper left and upper right we conclude that alignment (i.e. possible flux non-conservation) cannot be the issue.
Bottom right: This shows that the scatter on bootstrapped albedo determinations are now larger than for sequential access, and that levels between 0.2 and 0.3% can be reached, at best.
So why is it 'bad' to pick the images in random order, rather than sequentially?
Speculation department: If it turns out that alignment simply isn't the issue then what? We have the shutter and we have the bias. In the above we assume that shutter works fine and that the bias is the same in all frames. We test here the shutter, by looking at the total flux of each image in the 100-image stack: Top panel shows percentage deviation of the total flux in each bias-subtracted frame relative to the average of the whole stack. The standard deviation is 0.46%. Flux is not constant from frame to frame. Why? Second frame shows the maximum flux in each frame after smoothing. The two graphs are very similar. The last frame shows the mean of a dark corner in each image. The graph does not look like the other two - so it is unlikely that strong sky variations or changes in bias caused the changes in total flux.
We guess that the changes in flux is due to either changes in atmospheric transmission or shutter variability.
But how could flux in-constancy cause the problems we see with increased errors when bootstrapping? Apart from a few big dips in the total flux, most points seem to be randomly scattered wrt the neighbours. In that case sequential access as opposed to random access of image sequence would not matter, would it?
There is the 'wobble' of the image as seen in a film we have on this blog - waves of 'something' wash across the lunar disc causing small transmission changes. Sequential images may then be more alike than images selected far apart, and alignment, being whole-image shift based, would not repair the problem due to 'wobble'. A test suggests itself - look at fits to single images in a stack and the scatter these have. Then consider whether that is more or less than the scatter seen between n-frame averages of the two types.
CONTINUED EVEN LATER:
It was tested whether albedo fit scatter increased or decreased when integer pixel shifts and sub-pixel shifts were used: It turns out that sub_pixel shifts is better than integer pixel shifts in most cases, but that a straight sum (i.e. un-aligned) is best! NB: This is the case for the present choice of 100-image stacks. YMMD.
Post-Obs scattered-light rem.Posted by Peter Thejll Sep 10, 2014 09:30AM Below, we have investiagted the phenomenon of 'Mickey Mouse ears' - i.e. the tendency for theoretical lunar images to have too bright cusps so that the residual image ends up with a bipolar structure ressembling Mickey Mouse's ears. We have investigated what happens to the ears if a semi-empirical BS model image is used, based on scaling and clip-selecting the BS from the observed image. We showed that the ears went away.
Now we investigate what happens to the quality of the fit - for our goal is to derive terrestrial albedos with minimal error (both scatter and bias; bias we can live with as long as it is constant; scatter is a pain).
We selected 10 images that we fit with both semi-empirical BSs and theoretical BSs and tabulated the values for fit-parameters as well as some of their formal fit errors (output from LMFIT).
In this plot we show the changes, in percent, of some key parametrs and their errors, in the sense that the purely theoretical result is subtracted from the semi-empirical result:
Change, in %, of some key fitting parameters. (semi-empirical - theoretical)/(½*(semi+theor))*100.
A: albedo, is larger by several percent in the semi-empirical (S) fit, than in the purely theoretical (T) fit.
dA: error on albedo reported by the fitting routine, is mainly smaller in S compared to T.
alfa: the power to which the PSF is raised is less by several % in S compared to T. This implies a broader PSF in the final fit in S.
ped: the pedestal added to the whole image has both positive and negative changes.
dX: the shift of the model image in X direction needed to align model and observed image. Mainly smaller in S than in T.
cf: core factor on the PSF - has larger values in S than in T. So the PSF peak is 'pulled up' in S relative to the wings.
contrast: lunar albedo map contrast, has smaller values in S than in T. This is consistent with the 'pulled up' core which 'copies' detail better.
RMSE: the root mean square error in the area of the image used for minimization, is mainly smaller in S than in T.
So, what have we learned? The two formal estimates of errors - dA and RMSE are both smaller in S than in T, so S would appear to be a better way to go.
Taken together - the smaller errors (on A, and RMSE itself) in S compared to T is encouraging, but we really do not know how much closer to the 'true value' for A we are, just that we are probably closer - at least in terms of scatter, not necessarily in terms of bias.
Using the semi-empirical method compared to the fully synthetic method adds very little in terms of computational overhead, so we will use S henceforth.
We note that the width of the distribution of albedos in units of the mean is 8 percent smaller in S than in T. The S method gives tighter distributions of albedo. Of course, the intrinsic (geophysical) scatter in albedo is present in this number too.
Post-Obs scattered-light rem.Posted by Peter Thejll Sep 08, 2014 11:30AM It seems our synethtic model images of the Moon have a distribution across the lunar disc that does not match observations. When we fit such an image to the 'band' near the photo-equator and subtract the fitted image from the observed image we get a residuals image with 'Mickey Mouse ears' - i.e. the residuals in intensity are bi-polar.
We are testing a semi-empirical method where the BS of the synthetic image is replaced by the observed BS, scaled to the right value. When such an image is fitted to the observed image we get no Mickey Mouse ears problem:
Left image: Residuals after a well-fitting synthetic image has been subtracted from the observed. Note the polar nature of the residuals - the cusps are clearly too bright while the area between the cusps is too dark. Right image: residuals when a well-fitting image, based on the observed BS, is subtracted. There is very little large-scale structure now.
The two fits deliver terrestrial albedo as one of the fitting parameters. For the left image (i.e. the entirely synethtic model image) we get A=0.37298, for the semi-empirical image we get A=0.36311, that is a difference of 2.7 %, which is considerable, given our science goals of 0.1%.
Whether the fit has been improved where it counts is a different matter. The RMSE for the two fits are identical (RMSE=0.026).
The large change in fitted albedo is a (regrettable) feature of our fitting method - change something a little bit and the fits change by several percent.
We suggest that a meaningful test of fit quality will be to fit the ensemble of images with the two methods above and see if the scatter is less in one than in the other. TBD.
Data reduction issuesPosted by Peter Thejll Aug 30, 2014 11:43AM We have commented earlier that for an otherwise well-fitted model image the 'horns' or cusps of the fitted image appear relatively too bright compared to middle-of-the-disc pixels. We investigate this now.
For an observed image and the corresponding best-fitted model image we extract certain pixels defined by the incoming and outgoing angle wrt to local normal on the lunar disc and compare the intensities in the observations to those in the model image. Here is an example: The upper panel shows the observed image with certain pixels picked out (the even dark band along the terminator). These pixels all see the Sun under angles between 55 and 65 degrees wrt the local normal. In the lower panel we show the intensities of those pixels in the observed image (black) and the model image (red) plotted against the angle between local normal and the direction towards the Earth.
We see that the model 'dips' for large angles (i.e. when the direction towards Earth is almost paralell with the local surface) while the model values (red) are unwaveringly high. We also notice that the model values start low at low angles and rise with a modest plateau starting at 75 degrees or so.
If we repeat the above for other sun-angle bands we see the same pattern - model is 'flat', observations 'dip' when the angle to Earth is large, leaving the cusps always brighter in the model than in the observations.
From flux to AlbedoPosted by Peter Thejll Aug 11, 2014 10:29AM This is an image of the Moon occulting Saturn. Saturn is evidently brighter than Earthshine - however, this is a phase near Full Moon so the Earth is near New, and thus faint. Still, eh? From APOD. Image by Carlos Di Nallo.
Exploring the PSFPosted by Peter Thejll Aug 01, 2014 01:36PM With a 1-degree occulter I took an image of the Sun. A slice across that image, through the darkest spot on the occulter gives this profile, plotting from darkest spot and outwards:
A log-log plot of the radial profile of the occulted Sun, on August 1 2014, using f22 and 1/500 s exposure, and a 1-degree occulter. Pixel scale is 0.04 deg/pixel, so 1 degree is at 25 pixels and the 200 pixels are at 8 degrees. The image is from the R-plane, G and B are not shown.
The occuluter is the dark part at left stopping near pixel 20. The drop-off after pixel 200 is some roof or something like that, while the curve between pixels 20 and 200 are what the halo around teh Sun looks like. The CCD is 12-bit so saturation starts near 4000, and non-linear response, if any, before that. However, we see clearly that the dropoff due to scatter in the optics is faster than dropoff of halo intensity. This implies that optical scatter plays a smaller role in defining the halo than does atmospheric scattering.
Note that this is with a multi-element SIGMNA telelens - albeit on a rather hazy day. Of interest for our own little earthshine telescope is to repeat the above on some crystal-clear day (hopefully approximating conditions on Mauna Loa) and see if the optics drop-off is also faster than atmospherics then.
On Mauna Loa, of course, the effects of optics are what they are independent of altitude while the atmosphere is MUCH clearer there - and there is less of it than in Frederiksberg.
Post-Obs scattered-light rem.Posted by Peter Thejll May 23, 2014 02:37PM In our fits of model-images to observed images we fit a model with several parameters. One of them is a 'sky pedestal' which is a constant that is added to the model image before fitting in order to accomodate sky brightness that is not due to the scattered-light halo.
We include this term because we figure that the sky might be uniformly bright for several reasons:
1) moonlight could be reflected from the ground up to the sky and back into the telescope 2) airglow could brighten the sky 3) unresolved stars and zodiacal lights shine from the sky behind the Moon 4) light could be scattered inside the telescope and contribute a more or less uniform 'haze'
We show here the magnitude plot of that pedestal term against lunar phase for our 5 filters:
Error: axis labels have been switched!
In order to interpret this plot know that the pedestal term is ADDED to the model image, so a positive value of the pedestal implies that the model is representing a sky brightness, and that a negative value of the pedestal implies that the model needs to subtract something before it can fit the observation well.
Notice that the pedestal term is negative for all but the VE2 filter. That is, only in VE2 is a well fitting model one that has a sky brightness contribution. For the remaning filters the model needs a subtracted term for the fit to be good.
Notice also that the VE2 pedestal is about 4-6 times larger than the other filter's, in absolute terms.
Notice also that the term goes to zero for large phases (i.e. near New Moon).
So - turning to speculation now - what is going on?
Only VE2 behaves as expected: As the Moon gets brighter more light has to be removed. This could be Moonlight scattered from the ground, onto the sky, and back to the telescope [See the paper by Bernstein, R. a. R. ∼A. "The Optical Extragalactic Background Light: Revisions and Further Comments." The Astrophysical Journal 666, 663–673 (2007). They model the moonlight reflected from the ground near a telescope.]. Or light scattered inside the telescope - but not airglow (why would it depend on lunar phase?). Nor can it be zodiacal light because although it depends on lunar phase when you observe the Moon in particular it has the wrong phase-dependence - towards Full Moon the ZL near the Moon is low; towards New Moon the ZL near the Moon is stronger (because the Moon is nearer the SUn and the ZL is strongest near the Moon).
So - the pedestal for VE2 behaves as we expect widely scattered light from the Moon should behave.
By the way, let us call this light 'ADL' from now on - ambient diffuse light, to tell it apart from the light scattered in the halo, which has a strong profile with distance from the Moon.
But for the other filters the pedestal behaviour is opposite!
I think this could be a sign of the fitting process having a problem - the method has to specify negative pedestals for something else to work. That could perhaps also be the case for VE2 but to a smaller extent than the conventionally expected scattering: if it is bigger for VE2 a contribution could still be taken out leaving a positive pedestal for that filter.
Do we have any reasons to believe in stronger ADL for the VE2 wavelengths than the rest? The VE2 filter is of a different nature than the rest - perhaps it scatters more light? This would match the phase-behaviour.
What could the role of airglow be? While the zodiacal light and the galactic background and stars come from behind the Moon and are blocked by this, the airglow is all over the image. What happens when we fit such images with just one pedestal term? Can the procedure go wrong in the observed way?
What do we know about airglow? Does it depend on wavelength?
Let us assume that: In the VE2 filter there is a lot of scattering, and the method of fitting deals with this by specifying a large positive pedestal. For reqasons not yet understood, there is a tendency to make the pedestal a little too small - butthis is not seen in VE2 because the scattering is so large. In the other filters the scattering is less and the error comes through as a small negative pedestal.
Work on understanding why the pedestal needs to be a small negative number for the other filters. Perhaps it is a consequence of some of the choices of frozen parameters in the model. We do freeze 'core factor' and 'rlimit', for instance. Optimally, the factors that need to be frozen should be done so at values that do not induce strange problems such as the present one.
Secondarily, find out if the small negative pedestal influences albedo determinations significantly. With the DS intensity being proportional to albedo we can see that the small negative pedestal values directly correspond to albedo biases in the 0.2% range.
Data reduction issuesPosted by Peter Thejll May 23, 2014 01:12PM Inthisentry we estimated extinction coefficients for all data from each filter - that is, we did not study each night individually, but took all B, V etc data seperately.
We found, at that time:
B 0.15 mag/airmass V 0.10 mag/airmass VE1 0.08 mag/airmass VE2 0.06 mag/airmass IRCUT 0.12 mag/airmass
Since then we have eliminated some observations that we now know had problems of one sort or the other, and have the opportunity to re-estimate extinction coefficients. We now find:
B 0.18 mag/airmass V 0.11 mag/airmass VE1 0.06 mag/airmass VE2 0.09 mag/airmass IRCUT 0.05 mag/airmass
kB is 0.03 higher; kV 0.01; kVE1 0.02 lower; kVE2 is 0.03 higher and kIRCUT is 0.07 lower. The changes of +/- 0.02 are as expected given Chris' analysis of extinction from single nights, but the change in kIRCUT is large - however, it is now more in line with kVE1: the two filters are almost identical, so that is a step in the right direction.
We found these extinction coefficients by plotting extinction-corrected flux against lunar phase and fitting a third-order polynomial. For trial values of the extinction coefficient
Data reduction issuesPosted by Peter Thejll May 17, 2014 12:23PM In some of our images sky conditions were such that the halo obviously was 'too large' for good results to be expected. We seek to eliminate these images now.
One simple measure of a halo that is "too large" is by looking on the BS and finding those images where the halo drops slower than in other same-phase images. We calculate and index of halo-narrowness based on the ratio of brightness 20 pixels beyond the BS edge, and the maximum brightness in the image. We call this the Q20 index.
On nights with 'large halos' we also often see a bright sky quite far away from the lunar disc, on the DS part of the sky. We quantify this by the mean brightness of the 10x10 box in the lower DS corner of the bias-subtracted image.
We plot now the phase vs Q20 and the sky brightness vs Q20:
Each row is for one of the 5 filters, left column is Q20 against the illuminated fraction of the lunar disc (i.e. phase). Right column shows sky brightness on the DS vs Q20. Q20 is the ratio of the BS intensity 20 pixels from BS edge on the sky and maximum brightness of the image.
We see that Q20 is quite constant against lunar phase, implying that we are correctly normalizing the halo. We see only a small drop in Q20 for (perhaps) V and VE2 filters as phase becomes very small.
We see some outliers where Q20 is high compared to most of the data at same phase.
We see (right column) that sky brightness and Q20 are related - for large values of both, but otherwise Q20 is a constant.
In fitting images to models - discussed abundantly elsewhere in this blog - we realize the need to concentrate mainly on images with as little halo as possible, so we wish to eliminate the images with a large Q20 value. Where shall we set the cutoff?
On the plots the dashed line shows 1.3 times the median Q20. We (arbitrarily) choose to get rid of all images with Q20 above this line.
We also note that Q20 is largest for VE2 and smallest for B - light is scattered further the longer the wavelength is. Does this tell us anything about the scattering mechanism? I'd venture that if the scattering was basedon the Rayleigh mechanism we should see more scattering in th eblue - wonder how Mie scattering goes? At least some of the scattering is in the optics and is fixed and unrelated to the atmsopheric effects - perhaps that can be confirmed by considering some optics?
What of the sky brightness?
We see that VE2 and V have higher sky brightness values than the other filters - VE2 so more than V. Most B, VE1 and IRCUT images have sky brightness below 1 (count), while VE2 has mos between 1 and 10. V has most between 0.,1 and 1 butr a fair fraction above 1.
For VE2 we spculate that the sky brightness is realted to long-wavelength flux from the sky - this could be from water vapour - so we suggest that VE2 sky brightness is compared to MLO weather data for humidity. Since the H2O may be high up in the atmsophere and not a ground level where the hygrometer is, this may not be easy. Satellite imagery for the nights in question could be brought to play on this issue. [Student Project!].
We are uncertain why V should have sky brightness issue if nearby B and VE1/IRCUT do not.
We arbitrarily suggest to also eliminate all images with sky brightness above 2 counts (10 for VE2).
Removing such images from 'Chris list of best images' we are left with 494 in number, and the list itself is available on request.
For our observing times we extract all available met data and inspect them:
Left: (top) Vertical temperature gradient (T2m - Ttop) in degrees C against fraction of Julian Day (fJD). fJD=0 at midnight, so each night's observing starts at 0.8 or so and proceeds to the right, wrapping around at midnight. T2m and Ttop are temperatures measured on the MLO met tower at 2 m above ground level and at tower top (much more than 10 m up!). (middle) Relative Humidity in % against fJD. Ignore bottom panel. Right: (top) Histogram of vertical temperature gradient for all observing times (black) and for every hour of 2012 (red). (bottom) Histogram for RH.
We note that the vertical temperature gradient for our observing times almost always are negative - that it is warmer higher up than near the ground - this implies vertical stability. The opposite condition certainly occurs during 2012 but does not correspond to observing times - probably because conditions were bad (convectively unstable or cloudy?). We also see that the gradient becomes less negative through the night implying cooling and equilibration of temperatures. Changes in RH are small. RH during observations show a skewed distribution relative to what was available - so something about observing picks out not the driest nights.
We shall see if the few nights with large RH (say, above 40%) tend to correspond to particularly bad conditions, by inspecting observed images and checking for 'drifting wiggles' and plain old 'large halo'.
An example of what our fit residuals look like. Note the structure around the BS (right, on the lunar disc). Also the straight-edge structure on the BS sky ... do we have internal reflections going on here?
Structure on the DS is in the tenths of counts range.
Here is the same image, but now in % of the local value ... don't scream! We have +/- 50% errors on the BS - but less elsewhere ...
The dashed line shows the fitted V-band albedo of the 101st image, which is the average image made up of the aligned and averaged 100 images in the stack.
In fitting the above 100 images we found residuals that were 'wavy' - probably the waves of 'something' that is seen in the movie here. The residuals look like this:
If you fit the 100-image average image, you get this:
The residuals are much smaller - but also 'wiggly'. The RMSE of the 100-image average fit is about 3.7 times smaller than the single-image fit's. Can this, and the width of the histogram (1.8%), be used to form a scaling argument so that the uncertainty on the 100-image fit can be deduced?
which would give us an uncertainty on the fit of the 100-image average of 0.5%whichis quite a lot. In contrast, the standard deviation of the mean, SD_m=SD/sqrt(N-1), is 1/10th of the 1.8%, or 0.18%. This is more in keeping with the experience we have had with fitting ideal images with added noise.
The advantage of the fit of the 100 individual images is that you get the histogram and thus a measure of the actual uncertainty - when you fit the 100-average image you just get the fit - formal uncertainty estimates depend on knowing (independently) the noise on each pixel. One way around is to consider Monte Carlo Markov Chain fitting using the Metropolis-Hastings algorithm - this iterates the fit in a Bayesian framework and generates histograms of all fitted parameters, but is very very slow because MANY iterationsa re needed. Another way would be to bootstrap the single images, somehow.
Here are the similar histograms for the same night from V, B and VE2. Again, the dashed line is the albedo found in the 100-image average image, while the histogram is made up of 100 single-image determinations:
We now have 3 stacks for which the mean of the histogram and the albedo of the mean image are very close - this encourages us to think that the uncertainty of the determination based on the mean image is similar to the stanmdard deviation of the mean for the histogram. Let us test this by fitting even more stacks and their component images alongw ith the mean image. It would be a way towards using the mean-image determinations as the 'real thing' and designate the variations we see from stack to stack as geophysical and not based in analysis method bias.
Showcase images and animationsPosted by Peter Thejll Apr 23, 2014 09:26AM Here is a movie of what 100 images in a stack, on a VERY CLEAR NIGHT, looks like if you look at frame-to-frame changes, and histogram equalize the images:
If you monitor 4 patches of 20x20 pixels each (one on the dark side of the Moon, one on the dark side sky, one on the bright side of the Moon and one on the bright side sky), you get these mean series:
There is some food for thought here. Consider the DS patch - it varies by tenths of counts (this is mean over a 20x20 patch) - the mean of that patch is typically several counts - so we are looking at several percent to ten percent changes with time here. Of course, we do not fit individual images, but only the stack average.
Notice how BS sky patch is dropping in intensity in this sequence - this is consistent with BS patch getting brighter - less light is being scattered to sky here.
If the sequence of images is scaled to the total flux of one of the images in the sequence we get a small drop in BS patch variability - it goes from 0.22% to 0.12%. SO either the shutter is variable and causing this, or the total amount of light in the 1x1 degree frame is variable, for other resons: scattering or extinction may remove light from the beam entering the telescope. But certainly, the light in the small 20x20 BS patch varies on its own even in flux-normalized images, so non-shutter variability is present.
Real World ProblemsPosted by Peter Thejll Apr 21, 2014 09:58AM If we take a stack of 100 images, calculate the total counts in each fram and tabulate this, and then take the list of totals and compare the mean of the list to its variance we find that variance is MUCH more than the mean. This should not be the case if the totals were Poisson-distributed.
We find that variance is 100s to 1000s times bigger than the mean of the list.
This could possibly be due to shutter variations - the shutter is ... less than fantastic ... after all.
Let us inspect the problem and see if it was less at the beginning of operations at MLO. Perhaps wear made it worse as time went on? We did collect 50000 images or so.
Perhaps atmospheric transmission changes on short time scales is this big? Each of the 100 images in a stack took about a second to capture. Does sky transparency change by a lot over that timescale, on a 1x1 degree field?
Post-Obs scattered-light rem.Posted by Peter Thejll Apr 20, 2014 09:00AM In this entry below we suggested 4 things to try to improve the accuracy and precision of our fitting method. A fifth idea is that byt fitting small, sharp-edged regions on the DS we are allowing small shifts in the image to 'pull' thge fit. A similar comment applies to the effects of atmsopheric turbulence. We have elsewhere shown that the lunar disc 'wiggles' when you look at an animation of 100 images. Some regions (5-10 arcminutes in diameter) shift sideways by what looks like a large fraction of a pixel, relative to other parts. If we use small patches to fit on quality will suffer if we fit across a wiggly patch's edges.
Using larger patches might therefore be an idea, and we test that here, while keeping all other thinsg fixed. That is, we use the exact same pixels as before but instead of concatenating 11 row-averaged bands we average the 11 bands into one profile. Again, we use the set of 10 ideal images to which realistic noise had been added.
mean albedo: 0.28031400 std. dev. of 10 fits:0.00021 SD expressed as percentage of mean: 0.08 %
Compare this to what we found before, using concatenated strips:
mean albedo: 0.28051 std. dev. of 10 fits: 0.00030 SD expressed as percentage of mean: 0.11%
It would seem we have about 30% less spread in the values found. As before the known fixed albedo of this ideal image was near 0.2808, so we have moved slightly away from this, and the bias is now about 2 SDs.
The use of averaging the 11 bands into one (they follow the contour of the lunar disc) does seems to ensure that flux is not lost outside a narrow band. We still use radial bins (5 pixels wide) so flux can be moved from bin to bin that way. We hesitate to make larger bins, as the degrees of freedom (surely?) would suffer. We have about 28 radial bins.
So, we have less scatter but more bias! Not sure this is a way it's worth to go. Alsao recall that thgese are ideal images - we are not sure what happens with real images. One day we might fit all images using the concatenated and averaged versions of the fitting method and see which has the smaller scatter.
We jave more suggestions from last blog entry to try:
1) Use more sky pixels 2) Use more DS pixels 3) Use BS edge 4) Average several images
4. We can always do and is independent of other method choices. 3 is scary, so I will try 1 and 2 next.
Experiment 1: Same image, with differing noise. =============================
We pick an ideal image, generated from JD2455945, and to this one image we apply a PSF convolution, and 10 times generate images with varying Poisson noise. We then fit these 10 images, and look at the distribution of results.
mean albedo: 0.28051 std. dev. of 10 fits: 0.00030 SD expressed as percentage of mean: 0.11%
The actual value for albedo used to generate the ideal image was 0.28083594, which is 1 SD above the mean value found.
So, we can (again) confirm that in principle, we can achieve fits with small bias and small errors, as per our original science goal.
Experiment 2: Same image with noise, different fit starting guesses. =========================================
We pick now one of the noisy artificial images and refit it 10 times - each tiome starting at a random guess near the supposed best guess. We pick our vatiation froma normal distribution with amplitude 0.03 and multiply the fixed starting guesses by 1+0.03*N(0,1) for the 8 starting parameters (there are 8 - 7 before - we added shift om image along y-axis also). These guesses were then allowed to converge and we collected the results. We found:
mean albedo=0.280183 S.D.=1.4181365e-05 SD in % of mean=0.0051
We see that the sactter is very small, i.e. that convergence is almost always to the same value. However, there is a bias in that the known albedo (see above) was not found.
The bias is (0.280183-0.28083594)=0.0006529 which is 0.23% of the correct value.
From this we lkearn that a) convergence may not always be dependent on starting point - the same best solution will be arrived at, and that b) the noise added to the image itself will have enough influence, even if small (i.e. realistic for averaging 100 images in a stack, and coadding 20 'lines' in each of 11 'bands') to cause a bias in this perfect-case example, where noise is only Poissonian.
The bias is a mere 0.2% of the correct value which we could lkive with given our science goal of 0.1, I guess, but reality adds more problems to our images than just Poisson noise. These effects appear to give us a typical spread of 1% - about 10 times what we saw in experiment 1, above.
As it is, we are using 1/10 of the pixels in the image for our fits so we have at most a factor of 10 more pixels to work on. This could in principle give sus a factor f 3 reduction in the uncertainty, if all pixels are equally effective at driving the solution. If all went well, our 1% errors coul dbecome 0.3% errors which woul dbe acceptable (just), in the absence of a bias.
What can we do along this line?
1) Many of the DS sky pixels are not being put to use - they coul dhelp set the sky-level of the fitted model better. 2) Not all DS disc pixels are being put to use, but there is a limit how far onto the disc we can go before BS halo starts being a too importnat factor, but perhaps we could double the number of pixels. 3) We only use the BS disc and BS sky to calculate the total flux which is used to ensure same flux in model and observation. We might be able to also fit some of the edge near the BS or just try to match the level of the BS sky near the disc - but the levels are much higher and our cirrent use of errors based on differences between absolute levels would become dominated by the BS pixels in driving the fit. 4) We can use more images from the same night - at least inside some period of time where the geometry of Earth and Moon does not change significantly - perhaps half an hour or so. In that time some 10-30 images could be gathered in reality. If several filters are wanted then perhaps 10 images in each filter (in practise it seems we could reach that on some nights, inside a half hour) and then repeat for other filters.
In summary, if erros were due to Poisson noise onkly, we could hope to double the number of DS disc pixels, use a lot more of the DS sky pixels, and perhaps work towards 10 images in each filter inside each half hour, giving us, potentially, the required improvements in uncertainty.
We can work on this!
There is a "however", however: All the noise is not Poissonian - there are issues having to do with:
a) Bias subtraction - and airglow - and zodical lights: letting the 'pedestal' solve for these may be a bad idea - perhaps we should fix the pedestal at predicted levels in each image? This needs some wor, since we are sensitive to errors there too! b) The atmospheric turbulence causes 'wiggle' in the images,. described on this blog, and this is a factor in driving the error in the fits. c) The model we fit is not faultless either - the BDRF is a simple one and we see hints of some azimuthal-angle problem ain fitting along the edge of the disc. Model couild be improved. d) Other parameters in the model might be fixed too - such as the 'core factor'. Experiments are still needed to see if this is a good idea.
Post-Obs scattered-light rem.Posted by Peter Thejll Apr 17, 2014 11:12AM After exhaustive (-ing?) tests of fitting the same image over and over again under varying conditions, we now turn to fitting all images in two colours from one night. We take B and V images from JD2456045.
We have before fitted 'lines' going from DS sky onto the DS disc itself, and then moved on to another position along the DS edge. Thsi helped us see that fits along the edge could vary systematically, and we have speculated that it was due to the BRDF-model used had some azimuthal angle-dependence that was different from reality.
Instead of getting several fits along the edge and combining these, we now choose to combine lines and make one fit, thus allowing the errors to balance out somewhat.
A typical fit, for 11 lines, looks like this:
The upper panel shows 11 concatenated 'lines' going from 50 pixels off the DS edge to 70 pixels onto the disc. The white curve is observations and the red is the model. The lower panel shows the difference between the two at 'best fit'.
We have 8 parameters in the model that together describe such things as the terrestrial albedo (the main goal of the exercise) and parameters that adjust the PSF, as well as the relative alignment of model and observed image.
In the above, the residuals are small, buts how systematic behaviour near the ends - going along the sequence of the zig-zags we are essentially sweeping along the edge of the DS - so something is not well at extremes of the sequence, but near the middle the residuals are very small.
We smooth the concatenated lines (and corresponding model line) before fitting - this helps reduce a problem we had before, namely, that repeated fits of the same image starting at slightly diffeent (1% different) starting guesses could finish up at slightly different 'best fits'.
For all images in B and V for one night we can compatre the various quantities determined in the fitting:
From top to bottom, left to right: Albedos in B and V images against time of day (not really Julian Day, but fraction of day since that JD started); alfa of PSF wings against time; PSF core radius (here held fixed at 3 pixels); pedestal against time; image shifts in X and Y; lunar albedo contrast factor; PSF core factor; and RMSE against time. Black crosses are B-band, while green crosses are for the V-band.
Here we see a clear difference in B and V albedo, with some scatter. The PSF is behaving diffrently in B and V (core factor, pedestal and wing alfa). Lunar albedo amp contrast factor is unaffected by B and V. RMSE is bigger for B than for V. Image shifts are near 1 pixel in each idrection.
The above choice of a fixed core radius was prompted by noting that leaving it free to vary inside the range 2-10 would cause the fit to pin the core radius to the lower limit in V and the uppe rlimit in B. We argued that some of the factors core radius/core factor/ andpedestal were degenerate and chose to fix the core radius at a compromise value. We note the extreme values of the core factor parameter for B, compared to V. In essence, a large core factor implies that most of the flux is in the core of the PSF and little is in the wings. It is hard to say if this is what we expect for B vs V - the colour-dependence of light-scattering in optics (and atmosphere) is not understood by us. Yet.
The observations in B and V were not obtained at the same times (because we use filter-wheels) so it is not easy to combine the B and V albedo determinations. Some of the scatter seems to be correlated (first points in left top panel above), but the rest do not. The means and standard deviations and standard deviations of the means, and the SD in percent of the mean, for B and V are:
So we have scatter on albedo determinations of near 1%, like BBSO. Formal errors on the determination of albedo for a single image depend on being able to estimate the uncertainty of the data. As we clearly have some systematics going on (plot at top of this entry) this is not possible.
For the night JD 2455945 (the subject of our paper) we have fewer good images, but get is:
Post-Obs scattered-light rem.Posted by Peter Thejll Apr 12, 2014 05:03PM We have been looking at the 'pedestal' which is a part of the model we fit to our observations. With a black sky and perfect bias and dark-current removal, the pedestal should be 0, and the fitted model should have a halo that sloped off from the Moon and perfectly explained all brightness seen on the sky.
In reality, there is background galactic light, zodiacal light, airglow and any terrestrial light (Moonshine reflected from the ground!!), to allow for. To this end we have operated with this pedestal in the model, which was a vertical offset of the model. In looking at fits we notice that the pedestal is degenerate with the halo-parameters in such a way that they tend to lie along a line when a lot of fits are compared, each started at slightly different points - even for the same image repeatedly fit. The use of this pedestal therefore leeds to a bias in the determined albedo. Which is not good.
We therefore have tried to evaluate expressions for the additional sky brightness so that the pedestal could be fixed at an assigned, physics-based, value and not interfere with the fit itself.
In doing that we had reason to revisit and elaborate on Henriette's error budget analysis. In particular, we performed the analysis allowing for error terms in DS and BS fluxes, dark currents, and exposure times. We introduced the uncertainty on the superbias factor and see now that it is the dominating factor when area-means are considered.
For area-means of 20x20 points, the error due to the superbias factor is as large as the error due to Poisson statistics on the DS. For smaller areas than 20x20 the DS Poisson error dominates.
Since levels of sky brightness are of the same order as the contribution due to factor uncertainty, or larger, we see that we have a problem.
We could try to estimate sky brightness better, but as we realize that fitted albedo varies, for a single image treated in different data-reduction ways (bias removal, and stacking), we need to also re-consider the use of 'anchoring' of the model halo in the DS sky part of the image. This is what BBSO achieve by fitting a line to the DS sky and subtracting it from the image.
Improvements in estimating sky brightness could also be pursued, but all constant contributions will disappear in 'anchored' procedures. The point is to not let the fit be biased by such terms.
Post-Obs scattered-light rem.Posted by Peter Thejll Apr 06, 2014 12:49PM We continue the investigation into how well albedo can be determined by looking at justa single image and applying various methods to it.
We compare the 'old' method to a new method. The old one consists of plotting row-averaged slices that start on the DS sky and proceed onto the DS disc. We place the slice at 9 different locations up and down the edge of the lunar disc. The 'new' method consists of averaging the whole image (and fitted model) into one row by averaging along columns. We do two runs of the old method - varying the distance we go onto the DS, and one run of the new.
Black crosses indicate the albedo found by fitting along 9 slices ono the DS disc going from 50 columns away from the edge to 100 columns beyond that. Red crosses indicate the same but going to 50 columns onto the disc. Green crosses indicate results found when column averaging the whole image but starting from 9 different starting guesses of the fitting parameters. The dashed and dotted lines give means and standard deviation of the mean for all trials.
We see that the three methods give a spread of values, with few repetitions. The standard deviation of the Mean is about 0.25% of the mean, but the standard deviation of the distribution is about 1% - single determinations will therefore vary a lot, while the mean over many trials can give a mean with small error.
We notice that the green crosses - which are results of identical trials in the 'collapse the whole image into one row' method but starting from slightly different starting guesses - all lie below the mean and have some scatter - so the new method is biased wrt the old one and the methdo itself is finding 'best fits' that depends on starting conditions. That the convergence is not to the same number each time show that the LMFIT method used reaches a stopping criterion. We have chosen max 500 iterations.
Neither of the above findings are reassuring - fitting method choices determine the outcome. Possibly, averaging over a large number ffo different methods would give us a result that is stable and near the truth.
In summary, we are looking at method-dependent biases of half and whole percentages, and errors of means near 0.2%.
It is not clear if the bias comes from lunar albedo-map and reflectance-model choices, or are stochastic in nature. THe effect of stochasticity is seen from the green crosses where all things are fixed, except the starting guess for the iterated model-fit.
A single 100-image stack of the Moon was selected, aligned (or not, see below), bias-reduced and averaged and then subjected to our model fit. The fit selects 9 21-row wide 'strips' cutting across the DS edge of the disc so that we have 50 columns of sky and then 100 columns of disc to fit on. The 21 rows are averaged in observation as well as model.
Applying this method we get 9 estimates of the terrestrial albedo. We selected three methods for centering stack-images:
1) Align the images in the stack once allowing for sub-pixel shifts, then average 2) Align iteratively, refining the alignment wrt an updated reference image, then average 3) Do not align at all, just average
In this figure we show the albedo determined from the three sets: The determinations are shown in sequence, separated by vertical dot-dashed lines - method 1,3 and 2 (in that order). The overall mean and standard deviation is shown
We see systematic deviations: all Method 1 values are below the mean, all method 3 values are above the mean and method 2 is a bit above and a bit below.
The mean of these values is 0.3047 with a standard deviation of the mean of 0.0014 (0.45% of the mean), which is not too bad! However, we could clearly do better if the method-dependency was not present. How to remove this?
Not aligning images at all risks comparing 'smeared' observations with 'knife-sharp model images. The smearing occurs due to image-wander during exposure and can be as much as a pixel or two.
Revision of our alignment methods seem in order, given the above. Possibly shifts by whole pixels only? This is at least conservative away from edges.
Note that method 3 (the middle set) has the least internal scatter. For this set the standard deviation of the mean is just 0.2% of the mean. That is pretty strange, actually, since this is the 'do not shift images before coadding' method. Hmm. Then again, perhaps it is not strange, and we are simply seeing that the likely most fuzzy image has the most stable fit - as the strips select different parts of the light-and-dark lunar surface to fit on, contrast causes the fit to wander if the image is sharp. But on the third hand, our method now allows for 'contrast scaling'. Hmmm, mark 2.
Shifting and co-adding involves two issues - the interpolation required at a given non-integer pixel shift may not be flux-conservative (our tests seem to show this is only a small problem). The shift itself may be poorly estimated. Our shifting method is based on correlation - it does not look specifically at such things as 'are the edges still sharp' - i.e. Sally Langfords Laplacian edge-detection method. The use of edge-information only is not clearly the best method since it is based on very few image pixels and thus suffers from noise. Perhaps a hybrid method can be envisaged - "correlations and maintaining-sharp-edges"?
Post-Obs scattered-light rem.Posted by Peter Thejll Apr 01, 2014 11:34AM Since the start of this project we have worked on a method that fits suitably-convolved amd scaled theoretical images of the Moon to real observed images of same. Below we show that the method has improved, by fitting a profile from the same night with two methods from 2013 and now.
This is a pdf file showing the three plots:
As you can see, the present method leaves residuals that are structure-free - this is not the case with the older methods.
We are therefore approaching a situation where all data can be fit pretty much perfectly by a multi-parameter model.
The model uses these fitting parameters: a vertical offset representing sky brightness, a shift of the image along rows, a parameter for the width of the PSF core, a parameter that vertically scales the core inside that width, a parameter that sets the slope of the wings, a factor that scales the lunar alnbedo map contrast, and the terrestrial albedo.
Since the results still show signs of a dependency on just which part of the Moon we are fitting on, we conclude that the BRDF functions we use - Hapke 63 - are not capturing the right azimuthal-angle dependence of the reflectance. This area is now where improvements are to be found.
If we succeed we shall actually be contributing to Hapke-type work, by providing empirically validated BRDF functions.
Exploring the PSFPosted by Peter Thejll Mar 15, 2014 09:21AM In this post below we compared our canonical PSF (raised to various power) to the PSF shown in Figure 6 of the AvD paper.
We have since re-estimated /below) our PSF from radii near 0.1 arc minutes to 13 arcminutes using excellent data from multiple images of the star MARKAB. This PSF is empirical for one specific night and does not need the raising-to-a-power that the canonical one does. We plot it as is on the AvD figur, trying now to re-create the twopanels of that Figure:
Right panel shows our PSF as surface brightness in magnitudes per area.
I'd say our empirical PSF (green line) looks like it is realistically modelled by the canonical one raised to 1.8 and still beats the AvD PSF (the crosses).
We extended our PSF beyond 13 arcminutes by a third order power-law.
We have assumed a few things: AvD speak of normalized flux - we take this to imply that the PSF is divided by its VOLUME, not its area. We assume that calculating the surface brightness involves taking the total flux inside some distance from the peak and dividing it by the area.
Exploring the PSFPosted by Peter Thejll Mar 14, 2014 05:49PM We have some excellent data from observing the star Markab, in Pegasus. We have 334 images at 4 seconds exposure, and each is bordered by dark frames of the same expsoure time - thus we can subtract bias and any dark current easily.
We did this, and aligned and coadded the many images. Then we inspect the PSF by cosnidering the intensity around teh star in concentric rings:
We see that the core of the PSF is about 2.5-3 orders of magnitude above the point where the wings start. We see the wings drop off and join the sky level. The red crosses are median bin values. The blue curve is a polynomial fit, the coefficient of which are given in the table.The polynomial is found for a fit to 1/radius against PSF for the wings, and a Voigt function for the core out to about 1 arc minute.
Some points: The curve is not the PSF - because there are remnants of sky brightness - about 0.88 counts [star scaled to 50000 counts at the peak]. The PSF would be well approximated by the above polynomial without offset (first coefficient). The part of the PSF from 1-3 arc minutes is roughly linear, as if a simple 1/r ^alfa power-law applies. This is not the case when we consider the above.
We might revisit the extended halo we see in lunar images with the above insight into the effect of a sky level.
This paper discusses the use of novel lens coatings in commercial grade lenses (Canon) and DSLR cameras to build a multi-lens, co-additive system. They discuss the PSF they have in their Figures 6 and 8 (8 is about ghosts).
We can compare the PSF we believe we have to theirs: Here we have used the same axis-scaling as Figure 6 of AvD. The three curves are our normalized PSF profiles for three settings of the 'power'. These are 1.8, 1.6, and 1.4 - the largest at the bottom. Remember that the 'canonical PSF' we use already has a power slope of close to 1.7 so with the extra powers above we reach 3.06, 2.72 and 2.38, respectively. We know that the power of 3 is the limit for the wings of a diffraction profile, so use of values of 'power' near 1.8 implies we think we are diffraction limited - in the wings. Actual fits to observed images tends to give powers nearer to 1.7 or a total power-law exponent near 2.9. The points on the plot above are taken (by eye) from Fig 6 of AvD.
They seem to have not only a broader PSF than us, but the 'wings' appear to be linear in lin-log: that is, their wide halo is not a power law as we infer, but is rather an exponential.
On the face of it, our optics are 'better' than theirs, but hold on:
1) we do not actually measure all of our PSF - we infer that a power-law tail is in order; we only have actual profile measurements from point sources such as stars and Jupiter (almost a point source) inside several arc minutes. The rest of the wings are inferred from how the halo around the Moon looks at distance.
2) In their Figure 8 an image of Venus and a star is seen - and a ghost of Venus. The authors state that "the ghost contains only 0.025% of Venus' light". I think that number is so small that our requirements of 0.1% accuracies in photometry would be met. Note that ghost and the PSF are unrelated - the PSF wings describe light scattered while ghosts are reflections off optical element surfaces. The paper says that it looks like not all lens elements were coated with the novel coating.
What do we get from this?
I would like to investigate the PSf we have some more - this can be done with repeated images of a point light source in the lab: this would clarify the non-atmospheric part of the PSF. Despite current problems with FWs we may be able to pull this off with what we have.
I would like to understand where the (inevitable) ghosts are in our system - did Ahmed park them on the Moon itself? If they are same size as the Moon and well-centred they may not cause any damage since they merely replicate the image information (if same size and in focus).
It seems the AvD system has very weak ghosts but larger scattering than us (if we understood our PSF wings correctly) - if the weak ghosts are due to these new lens coatings then they are of interest to us in possible future designs of the system. If the scattering really is as high as it looks, compared to our (guesstimated PSF wings) then we have a better system than them, period. Mette commented that these coatings can cause scattering - wonder if the AvD people chose the coatings to lower ghost intensities or to get rid of scattered light? They do not say so directly, but do compare to large telescope PSFs (their Fig 6), and find less scattered light in their own system.
I need to undersatnd how they measure their PSF into the wings?
Observing logPosted by Peter Thejll Feb 28, 2014 02:07PM Tested a script while looking through the telescope. Script was set up to switch filters and take sequences of images.
Everything seems to be badly scrambled! Colours of filters do not match what is requested, and the sequences of images received are incorrect - for instance, when asking for 10 images at 2 seconds each I get about 3 images at 2 seconds and the rest are rushed through rapidly.
Shutter opens at strange times after a sequence.
It seems like the system is badly shaken up now. I think some of the above behaviour may explain some of the odd image-sequences we had at MLO, but the majority from MLO were good: now it all seems random.
My guess is that the LabView software has been jumbled somehow - it is easily done since the coding is not deterministic as text code is - it is all little coloured lines and boxes. It is easy to pick something up and drop it down in the wrong place. A test and a fix for this might be to restore an older version of the software - for instance something after Ingemar fixed the bad coding problems we found while at MLO and before the system was hit by lightning.
Another possibility is that hardware has been damaged in such a way as to scramble/ignore software instructions. Not sure how to test for this - we used a 'sniffer cable' to listen to the erroneous traffic to the mount, but I am not sure what to listen in on here?
Wonder if there is a way to 'reset the hardware'? Perhaps the FW are 'off by one' - some latch is not clicking home?
Inside the 'breakout box' there are lights to watch as a script runs. There is a light on a 'hager' relay and it evidently lights up when the two Thorlabs filtwer-wheel controllers are being adressed. They have displays that hsow numbers and I am guessing they show the position of the FW.
While running a sequence through the 5 filters I saw the numbers change. I think I caught a rythm corresponding to several passes through the 5 filters, but I swear there was an irregularity [hard to prove afterwards as there is lots of uncomfortable crouching and waiting to do]. Also strange is that the FW displays show two sets of numbers first, these then change to another set, then the hager light goes out and you can hear the shutter start working [its progress is ueven, however, despite being asked to take 10 identical images]. Here is a sequence:
where "A->B" means that the FW display first showed an A which then changed to a B, after 5 secodsn or so. Notice how FW1 alwasy goes "2->1". FW2 seems to be cycling all numbers. Notice how the start position in one par is alwasy 1 highe rthan the end position in the previous set. There is a cryptic sentence deep insdie Dave's LabView code about how Thorlabs uses one set of indices and his code another - this coul dbe it, I guess.
I think FW1 is the ND filter and is always set the same way - in the above test I was asking for the 'AIR' position and I think this is what we see. I will now run a test where I ask for one of the ND filters instead and we will see if the numbers change!
OK, I did that, and it seems I am right. FW1 is the ND filter. I set it to ND2.0 and the sequence 2->1 above changed to 5->6.
Should run a sequence now of just changing the ND filters.
Ok, did that too. This time I used a scriptthat always set the 'B' filter, but cycled through the available ND grades - that is AIR, ND0.9, ND1.0, ND1.3, ND2.0. The typical sequence of readouts from the Thorlabs controllers was:
we see how the colour filterwheel (FW2) always sets the same filter and always seems to start at the one numbered one higher. Visual inspection during the cycling revealed that the colour of the light switched between the colour of the lamp and Green - i.e. the FW2 settings actually acquired were 'Air' and 'V' - not 'B' as I asked for.
FW1 meanwhile seems to have cycled irregularly - 1,1,3,4,5, ... (2 is missing). It was not really easy to see whether various ND grades were inserted or not, but clearly AIR was one of them since I could see colours of the light.
Note that in a sequence "A->B" a sound - such as of a filter-wheel changing position - is heard before A, and before B.
Not sure what to make of this. It could be a signals problem where feedback from the actual position of the wheel is missed or missunderstood by the software so that new or wrong commands are sent.
Is the software that directs the FW giving absolute commands or relative ones, as in "Go to the blue filter" vs "go to the filter after the one you are at, which I assume is the IRCUT filter, so that you will arrive at the B filter since it comes after IRCUT"?
Why does the wheel change position twice during a set the FW-wheel command? Makes sense only if it always went to some home position, and then proceeded from there - but clearly it is not starting from the same position each time, given the readout above. Again, could it be that the intent is to send it to home each time but this goes wrong and it doesn't actually arrive at home, but assumes it did for the next relative command?
NEXT time, we need ABSOLUTE ENCODERS. I want a little man inside the telescope with a cell phone calling us with the ACTUAL position of each FW!
We disconnected the CCD camera so we could look through the telescope along the optical axis, at a lamp. We ran the camrea so that the internal shutter opened and closed regularly, and then manually changed the color and ND filters.
Strange behaviour was observed: Setting the color of the filter was not a simple affair - the colour would be a bit random, and the ND filter would reset - apparently by itself - after a few seconds.
We will try to run scripts next, to see if the command of the devices from the scripts are more consequential and whether the ND resets in the middle of a sequence. Our experience from masses of observations from MLO (before the lightning struck) is that stacks of images are all OK as long as the first image is all right.
Real World ProblemsPosted by Peter Thejll Feb 13, 2014 07:16PM An artist, Jeremy Sharma, in Singapore, has asked us what Pantone colour code the earthshine colour corresponds to. He heard of us through the Guardian story.
I can see two ways to answer him: One is via a conversion from colour temperature to Pantone code (or, equivalently, hexadecimal RGB code), another is to take a realistic image of Earth and averaging the RG and B channels.
I cannot, yet, find a conversion from colour temperature to Pantone code or RGB intensities, but the temperature to use is given by B-V=0.44 which is near 6400K, I think.
allows you to type in the above hexadecimal number and see the corresponding colour as shown on a computer terminal.
The above colour can be 'brightened' or 'darkened' by multiplying each of the RG and B values by some number (same for all three) and then converting to hex code. In doing this avoid saturating any of the three (they must be smaller than 255).
I did this so that a lighter rendition of the same colour would appear, and the result is
which you can also type into that page and see (they allow showing of two such colours next to each other).
Real earthshine is thousands of times fainter than moonshine so I doubt you can 'see' the colour if you make the numbers realistic (in fact, RG and B would be 0 in such an image since the numbers are integers and cannot be smaller than 1 without being 0). The two above colours have the same tones but differ in brightness.
I expect that a dye maker or a paint shop with a smart machine can mix your colour according to the hexadecimal code, instead of the Pantone colour code.
One estimate is provided by the 'profile fitting' where albedo is directly determined as a model parameter of the Sun-Earth-Moon system, and another estimate is given by the B-V determination we have from the cancelling halos.
From profile fitting albedoB/albedoV: 1.217 +/- 0.003 From B-V: albedoB/albedoV: 1.202 +/- 0.005
The error on profile fitting results is propagated from the standard deviation of several determinations of albedo in B and V and thus is 'internal error'. The error on the second estimate is taken from the paper and is 'internal' also - the dependency on error of the solar B-V value drops out.
The two estimates are similar but, given errors, formally different.
The fitting result is based on 6 fits in total - three in the B image and three in the V image.
From flux to AlbedoPosted by Peter Thejll Feb 09, 2014 08:20PM The amount of clouds on Earth changes from day to day, and as clouds reflect sunlight (with B-V=0.64) more clouds cause B-V of the earthlight to drift towards the solar value, while less clouds shows the Earth bluer. How much do we expect the terrestrial B-V colour to change day to day on the basis of changing cloud masses?
Inspection of GOES West full-disc satellite images of Earth shows that the local noon (i.e. disc is fully illuminated) brightness variations due to changing clouds amount to about 3.2% of the mean brightness. As the Earth below is dark (Pacific Ocean) almost all of the brightness is due to clouds so we are not too far off by saying that the amount of clouds varies by 3.2% around its mean value on a daily basis. With Earth on average being almost 67% cloudy we can re-use the Stam models of expected terrestrial spectra to see what the expected changes in B-V is due to cloud variations.
We find that B-V will vary, with a standard deviation near 0.005 around the observed value of B-V=0.44.
This is smaller than the total observational error we have. We have shown that the theoretical Poison-noise limited uncertainty would be 0.005 in B-V, but we cannot observe that well. Yet.
So - we should not expect to see values for B-V very different from 0.44, which helps explains why we get the same value as Franklin's mean value using just a single (but precise) observation.
We may be able to see 2 and 3 sigma deviations in the cloud cover, however.
The purpose of our telescope was never to observe daily cloud variations - in the long term we hope to be able to qualify that we can set limits to climate-change induced changes in albedo.
Also, the above is just an investigation into whether we can use COLOUR changes to quantify albedo changes - we still have to quantify how well our direct-photometry measurements can see albedo changes. Paper II.
From flux to AlbedoPosted by Peter Thejll Feb 09, 2014 01:33PM In an interesting paper by Antilogus, et al subtle effects of electron interactions inside the CCD material itself are discussed, which could be a tool for understanding and perhaps quantifying CCD nonlinearity.
We have tested our own Andor BU987 CCD camera for these effects and see them.
Plots of row- or column-neighbour means vs correlations. Top row: first panel: full set of values of means vs column-correlations
- we note the strong non-linearities that set in at about 50.000, and
Top row: second panel: for just the range of mean values below the onset
of non-linearity a robust regression is performed (red line).
Bottom row: same as top row but now for row-correlations.
We note the general correspondence to results in the 'brighter-fatter'
paper: rows are more strongly correlated with their neighbours than are
columns - the ratio of slopes is about 2.0.
Can this be used quantitatively to correct for non-linearity in our CCD? The camera linearity was already tested for linearity while in Lund. See Figure 8 in the report.
Observing logPosted by Peter Thejll Feb 04, 2014 04:31PM Testing the system on a lamp-illuminated flat field. With a script, sequences of images are taken with commands to change filters. Kinetic sequences.
There is obviously some problem with either the shutter or the setting of the FW, because while some images are well exposed at more or less recognizable exposure times (we use a different lamp from the hohlraum we had on MLO) some are simply almost all black at those same exposure times.
To test if it is the shutter that is failing to open, or the FW that does not set, we must eliminate one thing at the time - first let us run a script with exposure times of varying lengths, and simply listen for how long the shutter is open.
If it seems the shutter is always open as long as we ask it to be, we move on to the FW - perhaps we can open the telescope box and simply look at whether it turns (pant a white spot on the side!).
Bias and Flat fieldsPosted by Peter Thejll Feb 03, 2014 12:47PM Now that the camera is up and running again, a series of exposures have been taken to test the bias field.
Below is the old and new compared, and the difference in percent:
The bias level has changed by 1.3% or so, and the slope of the new field is different from the old - there is also some, smaller, change in curvature. The change in mean level is not so important since we scale the superbias to the current bias mean value anyway, but the change in slope is important. The relative level from up-slope to down-slope is something like 0.1% which is the adopted science goal for our project.
Further testing of the bias field will be performed as we proceed and if the above pattern is stable we shall simply start using a new superbias for currently acquired science frames. If studies show that the bias goes up and down and changes slope willy-nilly, as it were, then we have more to worry about!
Observing logPosted by Peter Thejll Jan 23, 2014 11:24AM As stated before, scripts can now be run and images gathered. We have solved a minor problem with disc-access that suddenly popped up. Still not sure what this is due to, but can now save images from automatic scripts. This clears the way for a row of lab tests that will start now.
Observing logPosted by Peter Thejll Jan 21, 2014 12:40PM It was possible to start a simple observing script today: Mount was not attached so nothing moved except shutters and FWs (and the SKE is still not working) - files were obtained but it turned out that something with the setup of permissions on the PXI has changed so images could not be written to the 'NAS' drive (a network drive in the instrument rack). Am seeking help from DMI IT department for this.
If that starts working we can run tests on bias fields at least, and later on flat fields (with a screen of some sort in the distance) and then focus tests and all this.
Links to sites and softwarePosted by Peter Thejll Jan 19, 2014 02:01PM Our paper, and the story in The Guardian seems to be generating quite a lot of activity on the net. Here is a collection of what I can find:
From flux to AlbedoPosted by Peter Thejll Jan 18, 2014 10:33AM In this entry we started exploring the Stam 2008 set of model Earth spectra. That entry was based on the 'land only' set of Stam models. Ocean-only models are also provided, so we can mix these with the land models to get more realistic Earth spectra. In Stam ocean models the albedo is assumed to be 0 apart from a specular contribution, so all colours are dominated by the Rayleigh scattering and a bit of sunlight reflected at the glint point. In land-only models the vegetation red edge has a large contribution, so we expect some changes when a land-ocean mix is introduced. We assume that Earth is 26 % land surface in what follows.
As before we calculate the spectrum of a range of cloudyness percentages. We find that:
B-V for Stam cloud free forest model is: 0.19 B-V for Stam cloudy forest model is : 0.54
B-V for Stam cloud-free ocean model is : -0.23 B-V for Stam cloudy ocean model is : 0.53
Note how similar the 100% cloud cases are for land-only and ocean-only, as expected. There is a small effect of the surface below the clouds (which is one of the features of the Stam set of models - others have ignored this effect).
Note also how blue a cloud-free ocean only Earth would be - this is, I expect, the B-V of our own blue skies on a cloud-free day, as we have discussed elsewhere. I think we should try again to measure blue-sky B-V with our DSLR calibrations.
As before we estimate slopes of the B-V vs cloud % relationship.
Change in % cloud cover per mmag change in observed B-V: 0.31 %pts/mmag Change in % cloud cover for total observed error in B-V: +/- 6.3 %pts Change in % cloud cover for internal observed error in B-V: +/- 1.6 %pts
The total B-V errors we have are +/- 0.02, mainly due to uncertainties in the Solar B-V as pointed out by Chris. If we leave all measurements relative to the fixed (but somewhat unknown solar B-V) we have internal errors of just +/- 0.005 mags and that implies we could determine Earth's cloud cover to within something like +/- 1.6%-points on the basis of observed colour alone. Not bad!
With a technique for observing B-V on all nights we could thus complement direct albedo-determinations (found with edge-fitting for instance). The colour-method has drawbacks and benefits relative to the direct albedo determination in that a difference is used instead of an absolute fit.
This is a point towards why we need to understand how the halos are formed and just why they cancelled on that night!
It is also fuel for fixing the SKE because with an SKE more images could be obtained where the remnant halos in B and V almost cancelled on the DS.
With the CCD working again, we have some hopes of understanding the SKE problems, as well as performing some very careful lab observations with halo due to optics only.
From flux to AlbedoPosted by Peter Thejll Jan 16, 2014 03:21PM Stam (A&A 482, pp.989-1007, 2008) has provided spectra of model Earths. From these we can construct B-V colours and compare to our observations.
Using the F0 and F1 model coefficients from Stam (cloud free and 100% cloudy, respectively) we combine these linearly as a function of cloud-cover fraction, and apply Johnson B and V filter transmission curves, and calculate B-V.
We check the method's correctness on the Wehrli Solar spectrum (we get B-V=0.64) and for Vega (we get B-V=-0.014). Both are very close to the accepted values, so we trust the method for other spectra.
We generate a set of models going from cloud free to 100% cloudy, and get:
Is it just my eyes or is this a really lousy rendition? I wish the blog software allowed for better pictures! Anyway, what we (almost) see is that the observed B-V for earthlight corresponds to about 40% clouds on Earth, with our observational uncertainty of +/- 0.02 mags corresponding to quite a wide range of cloudiness: +/- 8%.
Colours are thus not the way to determine cloudiness on Earth!
Conversely - the variations in cloudiness on Earth is considerable from day to day (at some phase angles +/-20%) so we should be able to use colour variations as an indicator of cloud variations, along with the direct albedo measurements.
The Stam model is based on assuming a Lambertian forest surface; Earth seen at phase angle 90 degrees, and polarization ignored.
Our (considerable) errors on observed colour is due to the conversion from earthshine colours to earthlight colours and the problems of taking the effect of the lunar surface into account. Our observational error on earthshine colour is 0.005 mags. This would allow us to determine Earth's cloud cover to about 2%-points from B-V colours alone.
A telescope in space (or on the Moon) looking directly at Earth could have colour errors that were even smaller.
The model has to account for orientation of Earth and so on, of course. This is not the case in the Stam model.
Observing logPosted by Peter Thejll Dec 18, 2013 10:30AM Sloooowly the system is coming back to us: We now have LabView Engineering Mode control of Front shutter, Iris, Mount, Filter Wheels, and now also Focus stage.
We still need to get the SKE stages working. We need to plug in the camera and see if it can be brought under LV control.
With these things running we are closer to getting some technical data from the system, such as bias frames, all-colour flat fields (for the first time), and we could study the shape of the PSF in different filters.
Important next steps are:
a) get the CCD working and under LV control b) Try running simple scripts again without telescope mounted
Future, ambitious, steps are:
c) assemble the actual telescope in a testing facility and run pointing tests - perhaps test long sequences of observing scripts indoors and actually debug the system so it would work under field conditions. d) See to the never-attempted integration with a weather station
Post-Obs scattered-light rem.Posted by Peter Thejll Dec 14, 2013 04:12PM At MLO the aerosol load of the atmosphere is measured and a database of hourly, daily and monthly average values for the absorption cofficient, in three optical bands, exists. Here is the link:
The data cover the period in 2011+2012 where we observed..
Our fitting procedures have produced estimates of the power a PSF has to be raised to in order for the convolution of model images with the resulting PSF to produce a 'good fit' at the edge of the DS disc in lunar images.
We compare our 'alfa' values with AERONET absorption coefficients, and get this plot:
We used daily average AERONET values, which goes some way towards explaining why the scatter for our data (the 'Power' alfa) is large while AERONET seems less messy. The overplotted red and green lines are two different robust regression fits. The slope is negative implying that large absorbtion coefficients correspond to small 'alfa' values. This matches how we empirically undersatnd the behaviour - on bad foggy nights (presumably with lots of aerosols in the air) we got broad PSFs with small values of alfa.
Repeating the above with hourly-average data the situation does not improve,
So our first interpretation of the above is that 'alfa' is only a poor descriptor of the aerosol load - 'alfa' is really just a dummy fitting parameter for us - alfa is the value that causes a good fit at the edge of the DS disc. It describes the shape of the PSF. Apparently thisis only weakly coupled to the total amount of absorption.
Observing logPosted by Peter Thejll Dec 05, 2013 02:17PM Got the FWs moving today, using very basic LabView modules. Engineering Mode does not seem to want to talk to its 'subVIs' any more.
in the VI 'set DO line' you can select port and line combinations. So far we know that:
port0/line22 operates the realys in the Breakout Box labelled RE2 and RE4. port0/line23 - RE3 och RE5
port0/line20 controls the Thor FWs. The two relays adjacent to the Thor controllers come alive. The FW can be selected via the pushbuttons on the boxes.
If we can get the camera alive next we could at the very least do lab work on the CCD properties - make color flat fields, study the PSF in various filters, and so on.
Observing logPosted by Peter Thejll Dec 04, 2013 04:21PM Today Hans was able to operate the front shutter and the front iris using LabView. The licensing problem seems to have been cleared up (it seems to have been a typo in a form somewhere - perhaps it happened at MLO, or later at DMI). The CCD camera and the Magma no longer are listed with 'yellow triangles' in the system - we have yet to plug it in and see it operate.
Next steps are to use same methods to get all devices running - Engineeering mode seems to be dead somehow, but all the 'sub VIs' are there and so far we have gotten the front shutter and iris VIs to work - can slew telescope from sub VI also.
From flux to AlbedoPosted by Peter Thejll Nov 04, 2013 01:13PM We use synthetic models of the Moon, illuminated by a model Earth, to generate fits to our observations. The properties of the theoretical models determine not only the fit itself but also the quality of the fit. Two important choices are made in generating the theoretical models - the lunar
surface albedo map has to be chosen, and the lunar reflectance model has
to be chosen.
We can choose surface albedo either directly from
the Clementine map, or as this map scaled visually to better match the
1970s albedo map by Wildey.
We can choose the reflectance models
in several ways - current models being tested are 'the new Hapke 63' and
the 'Hapke-X' reflectance model.
We test how well these choices work by considering the RMSE of the 'edge-profile fits'
generated between the theoretical Moon models and the observed ones.
Combing the two choices above in all possible wasy we get these results: There are four histograms here but pairwise hide behind the other. The major effect is that due to the choice of albedo map (blue vs green above) - the scaled Clementine map has 25% lower RMSE than does the un-scaled map.
The effect of reflectance model is very minor and can barely be seen
The RMSE is calculated from the profile of 20-row averaged 'cuts' at the dark side edge, starting 50 pixels from the edge and ending 50 pixels onto the disc. Fitting was performed vith the MPFITFUN routine in IDL, using a 4-parameter fitting model where intensity offset, horisontal pixel shift, terrestrial albedo and PSF-parameter 'alfa' were varied to obtain the optimal fit. RMSE was then calculated from the residuals between observed edge-profile and model same.
The Clementine map was scaled (see http://earthshine.thejll.com/#post253 ) against the Wildey map in the sense that the lower-resolution grid from the Wildey map was interpolated by the Clementine map and the scatter-plot generated. An obvious offset and slope difference existed and from the robust linear regression of one onto the other a scaling relationship for the Clementine map pixel values was generated and applied to all pixels in the Clementine map.
So, we may be able to help constrain the sort of work current space-missions do! We should not avoid to point this aspect of the work out, in the 'big article'.
Puzzling, at first, is the almost complete absence of an effect due to the choice of reflectance model. I can see effect clearly when total flux vs phase angle is plotted, so we simply do not have a large sensitivity to reflectance at the edge of the disc. Fair enough, only a local set of pixels are involved and the flux of the model bay be quite wrong while the fit at the edge is till very nice. So we should constrain our models by fitting many different aspects of our data - both the reflectance model against observed total flux as function of phase, and the albedo map by edge-fitting.
The subject of Clementine vs Wildey has been discssed in this blog before. Here:
From flux to AlbedoPosted by Peter Thejll Oct 17, 2013 02:27PM Hasinoff et al have (link) shown how images of the DS edge can be used to invert for the illuminating light field (i.e. earthshine) and receive low-resolution maps of terrestrial albedo distribution.The pdf file is here:
Some notes on this:
1) The edge pixels that can be observed from Earth are differently illuminated because the source of light is extended and at a slight angle with the observer's position.
2) Sally Langford described a method to find the edge of the lunar disk by using the Laplacian operator on the image and in essence do edge-detection. This method will be influenced by the fact that the edge is not uniformly lit. This applies, in principle, to our edge-fitting method too - except we have so low resolution that only 1-2 pixels at the edge show the effect. We do fit these - perhaps we should not.
3) We should be able to test the method Hasinoff et al describe from purely archival images found on the net - or we could try to get our own images (with the eyepiece projection method it should be possible to get high-magnification images of the DS edge). Or with the help of some willing amateur astronomer. Or by applying for 'service time observations' at some large telescope.
1) The albedos are derived as a scaling-parameter inside the Earth model we use. This model assumes the Earth is a Lambertian sphere with uniform albedo. 2) We use Hapke 63 to describe the reflectance of the Moon, and Clementine albedos for the lunar albedo. 3) We fit the DS edge - adjusting parameters in the synthetic lunar-image model so that a good fit is found between observed DS-edge profile and model DS-edge. 4) We see a 30-40% rise in albedo as Moon approaches Full (and Earth New). 5) We see a much weaker dependence of time-of-day in albedo. From considerations of the Opposition Surge (which may be inadequately represented in Hapke 63) we expected a drop in albedo towards end of day, and a rise at the start of day. We do see weak signs of the drop at the end of the day, but at the start it is ambiguous.
5) makes me think the problem is not the Opposition Surge in Hapke 63. 4) makes me think it is the total flux of the synthetic models that depend incorrectly on lunar phase. 1) makes me think that perhaps the Earth is not very Lambertian and that the discrepancy becomes much worse, particularly towards the crescent New Earth (Full Moon).
If scattered light is incorrectly included in the fit at the edge we would find a larger fitted albedo as lunar phase approached Full Moon, as we do - but there is NO WAY the fitting method is making a 30-40% bias-like error - it would be blatantly obvious when the fits were inspected.
What's going on? I would like to compare the observed run of total fluxes against the model-based expectation.
First we fit the edges of realistic but synthetic lunar images (made with Chris 'syntheticmoon.f' code):
Here models with albedo=0.30 have been edge-fitted - just like observations. We see that the edge-fitting method is completely able to retrieve terrestrial albedo.
So, there is a problem of some sort with the total flux of synthetic models. Let us look directly at the total flux from the synthetic images:
Here are shown (black) lunar model image magnitudes (adjusted to just about the right V magnitude at Full Moon) against lunar phase [FM=0], and expressions fro the same from Allen (red: 1955 edition, orange: 1975 edition), and from the JPL ephemeris system (green). JPL is based on Allen 1975 but adjusts fro actual distance to the Moon. Only ou rmodels allow for libration - hence the East-West assymetry in the black symbols.
It seems our synthetic images have a smaller range in total flux against phase than do the Allen and JPL expressions. By 1.5-2 magnitudes for some phase ranges, relative to Full Moon flux - or Full Moon is less bright by 1.5-2 magnitudes than are the Allen expressions, relative to intermediary phases. Our Moon images, relatively speaking, lack flux as we approach Full Moon.
In normalized-image edge fitting this will cause the model expression that is fitted to have been scaled by too small a number, making the step-to-sky at the DS edge too large, so albedo should compensate during the fitting by adjusting down. We see the opposite ...
Could it be that our near Full Moon albedos are all right? But they are so large - 0.4 and more at small phases: but what do we know? Perhaps we are making a relative error in estimating what the 'answer should be'?
Our observations have sampled the peak from just under 1 degree to about 1.5 degrees. From the plot in the above post it is clear that at JD start (noon in Greenwich, midnight at MLO) we sample the peak in the direction towards the peak (large to small phase angles). At the end of the JD we are sampling away from the peak (small to large angles).
The two situations are represented by Westerly and Easterly observing directions - Moon setting and rising, respectively.
There is a consequence of this 'daily sweep' of the range near the OS: as our reflectance model - Hapke 63 - does NOT contain the right OS we will see a rise or fall in the fitted albedo.
On the basis of the above argument I am making a PREDICTION: when we look at the fitted albedos we should see them INCREASE at the start of the day (because phase angles are dropping and to compensate the lacking reflectance the albedo has to go UP), and at the end of the day the fitted albedo will DROP.
Next, is a plot of the V-band albedos and the time of day:
We see albedos plotted against (JD mod 1). For each day one colour has been chosen, but some colours had to be reused. Each day has sequences of 3 points - this is because three fits of the model image to the observed image is performed.
To the left we see no slope, or perhaps a little rise towards the end now and then. To the right we see a general slope downwards. As predicted.
If the reflectance was correctly representing the Opposition Surge there should be no change in albedo as the day passed. We may be able to model the OS so that we can get the slopes to go away thereby determining the OS observationally. Troublesome will be the absence of a clear slope to the left, though. Hmm.
The large difference in albedo from day to day is mainly due to a phase-dependency in the determined albedos. The variations in terrestrial albedo are more subtle. We need to first understand why the fitted albedos depend relatively strongly on lunar phase. It could be due to problems related to model-fitting (although I doubt it since we essentially fit the step-size at the edge of the DS disk and this cannot be corrupted by scattered light). It could also be due to an incorrect reflectance function representation in the model used. This would not be the same dependence as discussed above related to near-opposition angles. The phase dependence I speak of is shown in the second row of panels below (the first being a repeat of the above plot): Above (second row of panels) each 'strip' is a nights observations. V-band albedos only shown. In this representation New Moon is towards the sides and Full Moon is at 0 degrees.
From flux to AlbedoPosted by Peter Thejll Sep 30, 2013 06:02AM We use Hapke's 1963 formulation for the reflectance function in making the synthetic models that are at the heart of converting observed earthshine intensities to terrestrial albedos. H63 has a part, at small phase angles, that correspond to the reflectances for the DS - on the BS, phase angles are large as the source is the Sun and the observer sits on Earth - but for the DS the source is the Earth. Seen from the Moon, the Earth is about 2 degrees in diameter - just what is the effective source-Moon-observer (i.e. phase) angle? For our roughly 500 observations we calculate the effective DS phase angle from geometry and a simple model of the sun-lit part of the Earth based on NCEP cloud cover, and geometry. The intensity-weighted distance between MLO and each sun-lit pixel is summed so that the effective photo-centre distance, in degrees as seen from the Moon, is arrived at. This is the plot:
Photo-centre phase angle for DS observations plotted against time of day. No correction for actual Moon-Earth distance have been performed; a standar distance of 384000 km has been used. Each linear sequence of points corresponds to an observing night, showing the evolution of phase angle as Earth rotated.
We see our observations distributed in a characteristic way: most observations early in the JD have decreasing phase angles; most observations at the end of the JD have increasing phase angles. The median phase angle is something like 1.8 degrees, with some as low as 0.5 degree and some as large as 1.6 degrees.
Since MLO is on Hawaii, opposite Greenwich in longitude, the JD change occurs for some observing nights in the middle of a set of observations - so the above plot should really be plotted from -0.5 to 0.5 instead of from 0 to 1.
We see that we are sampling H63 at reflectances near the 'opposition peak' and that the range of phase angles we have may sample enough that the errors H63 has in representing this part of the true reflectance function could be important in our work on reducing observed intensities to terrestrial albedos. We ought to have available several reflectance formalism so that we can see the importance during data-reductions. Hans has provided such additional formalism in the synthetic code, but is is complex to use (for me) so we must make an effort to get used to the new code, for the 'large paper'.
Data reduction issuesPosted by Peter Thejll Sep 22, 2013 11:04AM During data-reduction we found the many 'dragged images'. These are images where the shutter probably did not close before readout started. They look like this one (histogram equalized): The 'drag' is from the BS but is probably present everywhere, but not visible from the DS. The BS has a maximum near the right hand limb where the counts are near 10000. The 'drag' is near 240 at most. There is no gradient in the drag along the direction of the drag:
The absence of a gradient in the drag helps us understand its origins - it is the effect of constant-speed readout. The readout speed is supposedly 1e-6 s per pixel (in the image header: this may be read from the camera but could be a fixed number entered at setup ....). Reading 512x512 pixels should take 0.26 seconds. The exposure time for this image was set to 0.0274s. Being a 'dragged image' we have no idea if this is right.
I do not quite understand the ratio of maximum drag value to maximum BS value. If the drag/BS ratio is 240/10000 and the BS exposure time is 0.0274 s the drag must have lasted just 240/10000*0.0274 = 0.00066 s. The camera is a frame transfer system - is frame transfer that fast? Perhaps the 0.26 s readout time is a slower line-by-line operation? This page mentions 'a few msec' for the frame transfer: http://depts.washington.edu/keck/pg22.pdf . We have 0.66 msec, so - close.
If the shutter got into the act in mid readout the amplitude of the drag is affected by this. I think many dragged images we have seen are like the above one - dragging is seldom much worse than this.
Let us check other dragged images with higher exposures and see if the drag/BS ratio is constant or not.
The above was prompted by pondering whether we could have a 'shutter-less' camera. It would appear not - at least not if the drag is due to open shutter during readout. The readout would have to be 24 times faster than it is and I am not sure CCD cameras come with such fast readout. 24 MHz readout?
Post-Obs scattered-light rem.Posted by Peter Thejll Sep 22, 2013 09:02AM In our data-reduction methods, image-alignment is often involved. It is based on interpolation. An important questions i whether the mean is conserved during such alignment?
To test this we iterate the shift of an observed image, measuring the mean of a patch given by selenographic coordinates. We iterate many times - i.e. reuse the previously shifted image for next round - in order to get a clearer answer. Visual inspection shows the iterated image getting 'fuzzy' as the shifts recur. That is, image standard deviation suffers. How bad is the situation for the mean of an area?
The answer is 'not very much'. Iterating the shifts 40 times we find thatthe per-shift loss in mean value over a realistically sized patch on the DS is -0.005 %. The mean in the patch was about 10 counts which is 'bright' for the DS. At other lunar phases the counts may be just 2 or 3 - still just a factor of 5 from the above.
We would never shift the same image 40 times, of course, but the iterated shifts above allows us to build up a change in the mean that can be fitted with a regression. We have learned that one shift of an image may 'cost' 0.005% of the mean flux. This is a lot less than the goals for accuracy we have set, which are '0.1%'.
Added later: an identical test, but using ROT to rotate an image bya random amount, iterated, showed that errors per rotation were almost always less then 0.01%.
Post-Obs scattered-light rem.Posted by Peter Thejll Sep 19, 2013 04:21PM We are interested in just how good the method we use is when it comes to measuring B-V in small areas on the DS and the BS of observed images. In this entry we could find an error as low as 0.005 (once we stopped measuring incorrectly). Is this thelower limit?
We now use synthetic images with realistic levels of Poisson noise added to get the answer. We use 'syntheticmoon_new.f' from Chris Flynn and input the ideal image from JD2455814 - i.e. the almost Full Moon - and coadd 100 images, each with realistic noise added, and folded with a realistic PSF (alfa=1.7). We do this twice, with different noise seeds - convert to instrumental magnitudes, and then subtract the two images, pretending they are B and V images and call the result our 'synthetic B-V image'.
The resulting image is then measured in the areas designated by the agreed selenographic coordinates - those used in our little paper in Table 1. The errors found are:
0.0004 in the synthetic image 0.001 in the case of the real observed images
So, something is generating extra noise for us in the observed images, increasing the noise by more than a factor of two. This could be due to many things - real images are bias subtracted and flat fielded - synthetic are not; there is no readout noise in the synthetic images; image registration does not have to be performed (but could be simulated) in the synthetic images and we have seen variance do strange things during the necessary interpolations that take place during image alignment. There may be more issues to think of here.
This was for a situation with full illumination on both patches. What happens when one is in Earthshine only?
We will simulate this, but, for now, assuming the DS is 1000 fainter than the BS we would expect the error level to be sqrt(1000) larger in a DS-BS image, or about 0.012 mags. This is close to what we see (0.015).
Added later: Ona synthetic image of our lucky night (JD2455945) we find that the lower error limit on B-V BS-DS is
0.0021 and, for the observed image (see this entry) 0.005
So, again, we have a factor of two or so more nois ein reality than the best-case synthetic world predicts.
There is potential for improving our technique! For now we should report the above in the little paper. Fine tuning bias subtraction, flat fielding and image alignment - and understanding the role of image 'wigglyness' in alignment issues is needed.
Student projectsPosted by Peter Thejll Sep 14, 2013 04:43PM The Moon is occasionally present in corners of images of the Earth taken by Earth Observation satellites. Use such images and the known colours of the Moon to link the filter system used by the satellites to the Johnson UBV system. Use the resulting tool to calculate B-V colours of the Earth through time. Map out seasonal variability. Test sensitivity of the result to imposed variations in e.g. ice cover (by image manipulation) or similar for desertification.
Relevant papersPosted by Peter Thejll Sep 13, 2013 07:47PM In researching material for our first paper we come across the data in Allen for the colour index of Earth: "0.2". There is no evident source of that piece of information so in order to interpret it and use it in the paper we have to do some detective work. Here it is:
References given by Allen (various editions) in the table that gives B-V of Earth as '0.2'. ==========================
The references are both in the caption of the table, and in the B-V/U-B column of that table. The table is on page 299 of the fourth edition by Cox.
In the 1997 edition there is on p. E88 a table with a dash '-' for Earth's B-V. For the Moon there is 0.92 without a reference.
---------------------------------------------------------------------------- 2. Allen 1973 - the two colour columns of the table on page 144 of this 3rd edition - gives references to Irvine and Harris (see below).
In Allen 1955 there is on p. 159 a table with 'C' for planets and Earth has 0.2. In the notes it is explained that C is the 'colour index' and that it is offset from the larger published value by 0.1 to 0.2 ... What was 'C' as published and where and what is C compared to B-V? The references on p.160 of the 1955 edition are not specific for each piece of information, but we find as number  a reference to Danjon. There are two references, it seems - 1936 and 'Danjon, Comptes Rendus, 227, 652, (1948)'. The 1936 is an observatory publication on earthshine observations I am trying to find. The CR reference is about Mercury and Venus. Using information from Wildey's paper on Mariner 2 data for Earth where the mean Danjon C.I. of earthshine/light, BS Moon and Sun are given, we can estimate the linear transformation from the C.I. system to B-V. We get, using Wildey's data for Moon and Sun C.I. but not his B-V data for same (using instead Holmberg et al for Sun) that
B-V = .112+.671*.(C.I.)
With 0.2 for C.I. we get B-V=0.25.
We can test the effect of using B-V for Moon = 0.92 (an often cited value), this gives B-V= -0.0665+.8968*(C.I.) and C.I. = 0.2,0.3,0.4 gives B-V=0.11, 0.20, 0.29, respectively.
---------------------------------------------------------------------------- 3. Irvine et al 1968, AJ 73, 251, 807 Paper is on B-V for the planets - not Earth. ----------------------------------------------------------------------------
4. Harris, in Vol III, (eds) Kuiper and Middlehurst, p. 272.
The Harris chapter in the book briefly discusses the photometry of Earth - mainly in the form of a citation to Danjon's chapter in Book II of the series by Kuiper and Middlehurst. The Danjon phase law is given for earth, but no comment on colours.
---------------------------------------------------------------------------- Summary: The source of the '0.2' for Earth in Allen is thus not source 1, nor 3 - but 2 and 4 point at Danjon. The 1933 An. d'Obs. Strasbourg Vol 3 reference is in fact available online (now) at ADS. It is in French, and it is long - but towards the end things hot up because a summary of color index data for the Earthshine (Lumiere cendree) and Earthlight (Earth itself) are discussed, although in the Rougiere system. The effective wavelengths of this system are given in chapter 4 on p 171 and are
Rouge 606 nm Vert 545 nm Bleu 467 nm
so Rouge is between Johnson V and R, Vert is between Johnson B and V, closer to V, and Bleu is between Johnson B and V, closer to B.
Of interest is a description of the areas on the Moon used - it seems Crisium and Fecunditatis are used but only after 1928 is this fixed.
Also of interest are mean values (mean over several years) for the colour indexes, given on p 174 earthlight C.I. = 0.33 from -0.01 to 0.6 p 175 earthshine C.I. = 0.62 The variability is large and is said to be mainly seasonal. Note the addendum at the very end (p. 179) where everything important seems to be updated in a breathless paragraph! An updated value for C.I. for the Moon by Rougiere is given as 1.10 (cited by Wildey in the Mariner 2 paper), and the EarthSHINE C.I. is therefore updated to 0.64. There is a note that there isn't space to also update the earthLIGHT number but I assume it is upped by 0.02 also. So, if Allen is referring to Danjon's C.I. as 0.2 and states that it is 0.1 to 0.2 lower than the published value, we can see that Allen is talking, indeed, about the C.I. for Earth itself - earthLIGHT - and not earthSHINE.
So, using the linear equation developed above B-V = .112+.671*.(C.I.) and now inserting C.I.=0.35 (with limits at +0.01 to 0.62) we have for the earthLIGHT B-V = .35 (with limits from 0.12 to 0.53). For an earthSHINE colour of C.I. =0.64 +/- 0.3 we get a B-V range of 0.54 +/- 0.2 ---------------------------------------------------------------------------- Conclusion: This spans our observation as well as Franklin's, but with large uncertainties. Note the numerically identical B-V and C.I. values - so Allen is right in as much as replacing the 'C' from the 1955 edition with the 'B-V' of the 1973 (and on) editions. ----------------------------------------------------------------------------
Data to plot
Year (B-V)ds-(B-V)bs ------------- 1926-1935 0.54(pm0.2) - 0.85 = -0.31 to -0.11 1967 -0.17 pm 0.05 2012 -0.16 pm 0.02
Plotting this we have:
The Danjon data are an average of measurements from 1926 to 1935 and upper and lowe rend of the bars indicate the extent of seasonal variability, surmized by Danjon. The Franklin bars represent +/- 1 standard deviation based on the 12 determinates from 2 nights in his paper, and our error bars represent +/- 1 standard deviation based on scatter due to photon statistics in our measurement from a single night in two images.
Data reduction issuesPosted by Peter Thejll Aug 19, 2013 12:57PM In this post we commented on the problems we have with the requested and actual exposure times. It turned out NOT to be a good idea to correct the requested exposure times - certain colour problems got worse!
What happened was that B-V for the BS plotted against time of day showed a decided reddening towards the end of the Julidan day - this corresponded to the setting Moon. Airmasses were growing towards the end of the day.
When time of day and B-V were plotted we could see slopes using UNcorrected exposure times of 0.4 magnitudes/day. If we DID correct the exposure times as explained in the above link we got almost a whole magnitude per day in slope!
This got me thinking about whether the extinction corrections we perform are correct - they depend on knowing what the airmass is and knowing what the extinction coefficient is. If either, or both, are wrong we can expect colour-terms to pop up.
The airmass we have been using assumed surface pressure and 0 degrees C and 0 % relative humidity and a wavelength of 0.56 microns. More appropriate is the pressure that occurs at the MLO (we guessed it is 0.6*760 mm Hg), and a temperature of at least a few degrees C (we never observed in frost), and a relative humidity of 20%. Some of these quantities could actually be replaced by the available MLO meteorological data available - more on that later. We should also use an appropriate wavelength for B and V of 0.445 and .551 microns, for B and V respectively. However, using these better values will at most give an offset in B-V and cannot explain an airmass-dependent colour term. [or can it? gotta think about that one ...]
Next we must consider the formula used for the airmass. Looking in the IDL code 'airmass.pro' provided by Marc W. Buie, we see it is the "cosine based formula derived by David Tholen" with an update 2003. The formula appears to be: cz = cos(zenith[z]) am[z] = sqrt(235225.0*cz*cz + 970.0 + 1.0) - 485*cz When we have a reference for this I shall return to the question of whether the approximate formula is good enough.
Finally there are the extinction coefficients. Since we see an increase in B-V with airmass it is either the V magnitude that is being over-corrected or the B magnitude that is being under-corrected. Experimenting, I find that kV=0.08 and kB=0.17, instead of the 0.1 and 0.15 that Chris found from standard-star observations, givea a colour-free B-V progression with time of day (at Moon-set - Moon-rise is too underrepresented to be useful here). Is it within the error margins that kV and kB should have these values? Old man Chris says he estimated the values 0.1 and 0.15 'by eye' here but I think he also listed some regression results elsewhere, with errors - but I can't find 'em!
Anyway, using the values 0.08 and 0.17 gives a mean BS B-V of 0.898 +/- 0.054 (error of mean +/- 0.004). This value is relevant when keeping the discussion about the "van den Berg value for B-V=0.92" in mind, link here. Our own average of vdB62 data gives 0.88 +/- 0.02 - within the errors we have, easily.
ShuttersPosted by Peter Thejll Aug 18, 2013 08:22AM We had a method to measure exposure times - but the method depended on an LED inside the telescope system, and it appears that the system scattered that interfered with exposures. So we had it turned off. From data collected before the device was turned off it is possible to look at the relationship between requested and supposedly actual exposure times. For requested exposure times below 10 ms it appears we did not get anything useful. For requested exposure times above that there is a bias which decreases with the increasing duration requested. It is simple enough to fit a regression line to the above, it is:
and use this relationship to correct all exposure times. We have tried this.
On the left, results without correction of exposure time. To the right with.
We have extracted B and V magnitudes from the set of 'good images'. We have converted to magnitudes using Chris' NGC6633 and M22 calibrations. For B and V images taken within ½ and hour of each other we have calculated the mean B and V values for the BS and then B-V. We plot these values against lunar phase as well as the time of day, and we see that there is a phase- and time-of-day-dependence in B-V. When we compare B-V from images where the exposure times have NOT been bias-corrected with B-V from images where the bias HAS been corrected we realize that the problems are much worse for the bias corrected images, suggesting that the bias-correction is not valid.
That in turns suggests that the method for measuring exposure time is faulty. As far as we know the method was based on passing IR-light through the shutter aperture to a detector and then calculating the opening time of the exposure from data collected. We are not sure if this was based on a timing mechanism or a flux-collected method.
From our own images of a constant light source we ought to test the stability of the shutter. From the lab we have Ahmad's laser images, and from MLO we have many stellar images, Jupiter images and also hohlraum images.
Relevant papersPosted by Peter Thejll Aug 15, 2013 04:46PM Minnaert 1941 paper on the 'reciprocity principle' and other symmetry-considerations:
Photometry of the bright side of the Moon has been carrie dout for at least 100 years. An important paper is the one by van den Bergh:
He observed the Moon at 5 different occassions - two of them at Full Moona nd one of them at half Moon, and derived B-V colours for 12 regions, measured relative to Mare Serenitatis. He also observed standard stars and tied photometry of Mare S. to these - thus tying the B-V photometry of the 12 regions to the standards.
Here are some results and insights:
For Mare S. he found B-V = 0.876 +/- 0.022. Over the 12 regions on the Moon (20'' to 40'' large) he reports B-V relative to M. Ser. This offset has mean value -0.017 +/- 0.005, where the mean was weighted (by me) with the square of the mean error reported by vdB . Knowing B-V for M. S. we can say that an average over 11 regions on the Moon (I dropped one measurement as it was flagged problematic) is 0.859 +/- 0.023.
I would say that this is what vdB62 reports. From somewhere we havethe notion that he reports 0.92 for B-V - I do not see that in his paper. But he does quote Harris for saying that the mean over many regions on the Moon is 0.92, but does not comment further on that.
I notice that vdB does not allow for the different phases of his observations, and I think I know why - the photometry may depend on the lunar phase, but the colour does not (at least for these phases). We should check that, as we have BS B and V images galore!
Another paper, that by Wildey and Pohn:
contains masses of UBV photometry, and seems to discuss lunar phase and corrections thereof. Should be looked at. Many tables to digitize, though. Bachelors project?
We have some observationally based estimates of B-V for earthshine. We are working on various models of the same that would allow us to check our assumptions and set error limits.
A somewhat sophisticated model of the color (B-V) of earthshine has been made. It consists of a Monte Carlo simulation where scattering events are decided by random number generators and probabilities bazsed on albedos.
I draw photons from the wavelength distribution given by the Wehrli 1985 Solar spectrum. For each photon we allow a Rayleigh scattering event to occur with probabilities depending on the wavelength. If the photon is not Rayleigh scattered its further fate is given by factors describing the fraction of the surface covered by land (0.3), and the albedos of clouds, land (either Soil or Vegetation), and ocean. Some photons are absorbed while others are scattered into space. The scattered photons are collected and the emergent spectrum built up. If repeated for a large number of Monte Carlo trials a smooth spectrum results. The B-V colour of this is then calculated, using Bessel transmission curves for the Johnson B and V filters. The trials are done assuming that all land is covered by either vegetation or is bare soil. Cloud albedos are wavelength-independent, while ocean albedos are zero above 750 nm. Tables of albedos for land types and ocean is given by "Spectral signatures of soil, vegetation and water, and spectral bands of LANDSAT 7.Source: Siegmund, Menz 2005 with modifications." found at this link:http://bit.ly/14jD3qm.
The B-V of the incident solar spectrum is calculated as a check. The expected solar value of 0.623 (+/- 0.001) is recovered, where the uncertainty is the S.D. of the mean over the trials. The B-V values of the emergent spectra are subject to stochastic noise, so a number (20, each with 2 million photons) of trials is performed and the mean of derived photometric values, with standard deviation of the mean, calculated.
The results for earthlight are:
B-V of earthlight, assuming all land is vegetation-covered: 0.5692 +/- 0.00057 where the uncertainty is the S.D. of the mean.
B-V of earthlight, assuming all land is bare soil: 0.5759 +/- 0.00068 where the uncertainty is the S.D. of the mean.
The difference is 0.0067 +/- 0.00088 with earthshine due to the soil-only model being redder than the vegetation-only model. The difference in B-V is small, but significant.
A validating test is to look at how much flux is taken out at 550 nm - we know that extinction at that wavelength is near k_V = 0.1 so about 10% of the flux is removed. Since Rayleigh scattering is symmetric in the forward and backward direction (for single scattering events) we estimate that half of the light removed from the beam is scattered into space. Thus we expect the flux of Rayleigh scattered light to be about 5% of the incoming flux. We measure the Rayleigh/Incoming flux ratio to be 0.0521 +/- 0.0003. That is, as expected, about 5% of the sunlight is scattered by Rayleigh scattering into space, a similar amount being scattered downwards and giving us the blue sky, and totalling an extinction at 550 nm of near k=0.1. This assumes all absorption is scattering due to Rayleigh. At 550 nm that is not a bad approximation. For the B band at 445 nm we find that on average 2.3 times more flux is taken out by Rayleigh, corresponding to an extinction coefficient 2.3 times larger than in V. That is a bit steep, surely? Chris has measured k_B=0.15 from extinction data.
So, the model suggests that when you switch all land areas from vegetation only to soil only the B-V will redden by less than 0.01. The color of both modes is near B-V=0.57, with realistic land-ocean, and cloud fractions, as well as albedos. The model indicates a colour very close to the cruder model from the paper (B-V=0.56 at k=0.1). We use the exercise here to estimate that errors due to omission of land- and ocean- albedo is of the order 0.01 mags, and that B-V of earthlight is near 0.56.
Now, what did we find observationally?
We found DS B-V is 0.83. We also know that the sunlight has B-V=0.64 but appears as Moonlight at B-V=0.92. so that we can infer that one reflection on the Moon reddens light by 0.28 in B-V. The earthlight is also reddened by striking the Moon and we can infer that our observation of the DS implies earthlight that is 0.28 bluer, or B-V=0.83-0.28=0.55. That is very close to what our model suggests!
Note that all uncertainties here are governed by photon statistics -
not underlying physics or observational skills. Note also that we use
the phrase 'earthlight' when we mean the light scattered by Earth and
observed in space. By 'earthshine' we shall mean the color of the DS of
the Moon, which is illuminated by the earthlight. We have not even tried
to model the effects of hemispheric geometry, or any effects of
reflectance-behaviour on angles.
--------- added later -------------
This was done under an assumption that corresponds to 'airmass is 1
everywhere' - this is wrong, of course, since airmass for those surface elements near the edge of the terrestrial disc is larger than
1. It turns out that if you assume that airmass goes as sec(z) then the
area-weighted effective airmass for a hemisphere viewed from infinity is
2. We repeat the above trials with twice as much Rayleigh scattering.
The colours we get now are:
Assuming all land is plants : B-V = 0.5063 +/- 0.0009 Assuming all land is bare soil: B-V = 0.5152 +/- 0.00075
The difference is: 0.0089 +/- 0.0012
The Earthlight is now bluer than under the airmass=1 assumption (good), and the difference in turning land pixels from bare soil to vegetation is still small.
If we assume complete ignorance of the soil/vegetaion mix we get:
B-V earthlight, for airmass=1 : 0.57 - 0.58 B-V earthlight, for airmass=2 : 0.51 - 0.52.
The difference between airmass=1 (not realistic) and airmass=2 (realistic) is 0.06 mags.
I conclude that the model indicates a B-V of earthlight, at the most realistic assumption of the effective or average airmass involved in Rayleigh scattering, near 0.515 with a small dependence on wheather the land is vegetation or bare soil on the order of 0.01 at most.
I should repeat the above with VE1 and VE2 filters to see what change in NDVI vegetation index we might expect.
Data reduction issuesPosted by Peter Thejll Jul 24, 2013 11:56PM In using the fitting of model to observations at the edge of the lunar DS we have the following situation:
It is seen that the 'edge' of the observed image is not 'sharp' - it slopes over perhaps 3-4 columns on either side of the midpoint of the jump from sky to disc. The models often have problems matching that low slope and tend to be 'sharper' at the edge, at 'best fit'. We fit +/- 50 columns on either side of the edge so more pixels on the sky participate in driving the fit than does the 'edge-pixels'.
The shape of the edge is driven by convolution with the PSF - and the PSF core dominates in setting the sharp details. The slope of the DS sky halo is given by the wings of the PSF.
There is an opposition between fitting edge and halo slope - lower alfa would broaden the PSF and 'lift' the halo wings, increasing the slope at the same time as 'dulling' the edge of the model edge. With more pixels driving the fit at the sky halo the edge is bound to be ignored - thus too large alfa's probably are the result. The pedestal added to the model image to match sky brightness is competing with alfa's effect on the sky halo - part of the level of the sky halo is provided by the pedestal value while part is given by the effect of alfa on the PSF itself.
1) Exponentiation of the canonical PSF to power alfa may not preserve the shape of the PSF near the core in the required way.
2) More weight could be given to the 'edge pixels'.
3) Slope of the edge is not just due to rounding by the PSF - there is also some 'jitter' from co-addition of 100 frames so alfa is used to account for more than the effects of the PSF.
4) The fitted pedestal is sometimes negative - it should probably always be positive due to sky brightness after bias subtraction, but the bias varies by 1 count due to the temperature effect and is scaled, as described elsewhere. The pedestal is often near -0.1 counts - are we sure that the bias has been correctly scaled to the level of 0.1 counts?
Post-Obs scattered-light rem.Posted by Peter Thejll Jul 19, 2013 12:19PM Following up on this entry we test the Force method on ideal images. This should enable an understanding of the methods actual strengths and weaknesses.
We take an ideal image and fold it with two PSF's - one with alfa=1.74 and one with 1.76. The 1.76 image is the 'sharper' of the two, so we now subject this image to a test where it is folded with various PSFs so that it becomes 'less sharp', subtract it from the 1.74 image and look at the difference, expressing it as a percentage of the known intensities (known from the ideal image it is all based on).
Sharper pdf here:
In the upper panel we see the cross-section of the ideal image (black line) and the 1.74 image (red) and the 1.76 image (blue). In the lower panel we see the percentage differences. Black curve is for the difference between the red and blue curves (errors on the DS larger tan 1% everywhere, growing fast as we approach the BS), while the green and red curves show the percentage difference between the 1.74 image and a sequence of images formed by folding the sharper, 1.76, image with PSFs using alfa=1.98, 1.96, ... , 1.88. (top green curve has alfa=1.98, next down is red and has alfa=1.96, etc) The difference can, evidently, be less than 1% (for alfa=1.90) but not all over the DS at the same time - there is a tendency for the error to dive faster near the BS and in fact become negative. Near the DS edge to the sky it is possible to have errors in the 0.1-1% interval.
This test is realistic in that the PSF used is developed from real images; in that the procedure of modifying one PSF to another through the use of raising a canonical PSF to a power (alfa) is theoretically underpinned; and in that the brightness levels used in the ideal image are realistic.
The test is unrealistic in assuming that an image with one PSF can be transformed into an image with another PSF by convolution; the procedure of generating PSFs from a canonical PSF through exponentiation is probably only good in the wings of the PSFs.
The test shows that the method is not foolproof. Possibly it is better to first deconvolve an image and then forward convolve it in order to match PSFs - but this procedure needs to be tested on its own.
The test seems to show that for areas near the DS/sky rim a halo can be removed so that errors are in the few tenths of a percent range.
Post-Obs scattered-light rem.Posted by Peter Thejll Jul 16, 2013 01:01PM After some deliberations we now return to the 'edge fitting method' and its results. There are 19 nights on which more than 17 B and V pairs of closely spaced images exist. We consider those B and V image pairs with negative lunar phases, and look at how the B minus V (albedos, not colours!) values are distributed against lunar phase. We see this:
The error bars are estimated on the basis of error propagation in the B and V albedos from the edge-fitting uncertainty procedure. We have added small offsets in phase to get separation of points. The above actually contains data from 19 nights. To better see things we calculate the nightly median value Of B and V albedo, and now plot these:
Here, error bars are given by the standard deviation of the nightly values. Each point is labelled with the JD number. We see three night-after-night sequences: 2456073-76, 2456016-17, and 2456045-47. Given the error bars, these sequnces follow the same pattern - a rise in B minus V (albedo) as you go from new moon (left) to half moon (-90 degrees). Since the three sequences painmt the same pattern we are confindet in saying that 'Something Is Going On Here!'.
At the moment we do not know if this is geophysics in the shape of Earth albedo changing with phase - or some aspect of the halo (which increases from left to right above) being harder to model as phase grows.
Next it may be appropriate to study what the Earth actually looked like in the above three sequences. The negative phases selected for the above plots all correspond to the Sun illuminating Earth from Western Pacific/Australia/East ASia and westwards. As time passes on a given day the Moon sinks further in the West (as seen from Hawaii) and more of Asia contributes to the earthshine. On days in a sequence, at the same local hour, less of Earth is illuminated as seen from the Moon so the contribution to earthshine drops but is more and more 'sicle-like'. SUmmarizing:
For the negative phases selcted we expect:
a) during a single day - contribution to earthshine by continental areas increases, ocea contributions decrease, b) same time of day but consequitive days - earthshine contribution comes from areas closer and closer to the edge of Earths disc.
I think this means that - if clouds are evenly distributed - we should see reddening of the albedos during a sequence taken on one day, and a blueing in a series of days.
We see b)! Have we seen a)? Below is a plot of the B albedo minus V albedo values for each of the days in question plotted against JD day fraction.
sharper pdf here:
We see little evidence of any up- or down-turns in this. Perhaps 2456104 shows an upturn - i.e. the opposite of what we thought we'd see. The remaining narrow sequences above seem to show a slight downward trend as the day passes - i.e. reddening. Potentially, clouds dominated these days so that little of the surface was visible?
Post-Obs scattered-light rem.Posted by Peter Thejll Jul 16, 2013 09:28AM In understanding this this post, we now compare the results to those from the edge fitting method. We remind thereader that the Force method gives us the difference in B-V [magnitudes] between the BS and the DS, whil ethe dge-fitting method gives us albedos fro Earth in e.g. B and V bands. We look at the same range of phases in the two methods and repeat the relevant ploits here. First the edge-fitting method results:
Upper panel shows us B-band albedo minus V-band albedo against lunar phase (upper panel) and against the average alfa (B-alfa and V-alfa averaged).
We repeat the plot from the Force method here:
Here are the pdfs:
In the upper panels we see a slight dependence on lunar phase - B albedo minus V albedo as well as B-V [mags] do tend to rise towards Full Moon. Note that this implies opposite colours! The rise in B albedo relative to V albedo means that the Earth appears bluer as we approach Full Moon in the edge-fitting method, while larger B-V [mags] in the Force method implies a redder Earth as we approach Full Moon.
In the lower panels we see that the alfas used in the edge fitting method and the derived albedo differences have little relationship with each other, while the 'incremental alfa' needed in the Forcing method bears a strong relationship with B-V, as discussed before.
This seems to rule out the worst of the possibilities given in the Force posting - namely that if B and V albedos depended as strongly on alfa as B-V does then we would have a serious problem. In the present situation we are not in that position - but where are we then?
For both methods we seem to have outliers for phases less than -120 degrees. Let us take a closer look at these points. They are from night JD 2456073. The upper plot shows that the B and V albedos we could derive from edge fitting' differs by .1 between the upper group and the lower group. From the Force method we see that B-V [mags] differs by 0.5 between these two groups.
There are more data points for the edge-fitting method, in the plots above, than there are for the Forcing method. This is because more B and V combinations were picked tested for the former. The images picked were observed withing half an hour of each other. These images should be visually inspected.
[later:] Tried that - it gets messy: differences in the level-differences of the DSs in B and V can be spotted but do seem to depend on knowing the exposure times, and this is one of the things we do not think worked - better to trust the edge-fitting method, since it is a 'common mode rejecting method'.
Post-Obs scattered-light rem.Posted by Peter Thejll Jul 14, 2013 09:44AM In this blog-entry we discussed a method to force halos to cancel by convolving the sharper of two images in a B/V pair, until their halos matched and could be cancelled by subtraction.
From the many possible B and V image pairs that we have, a number has been selected, and the method applied. The criterion used was that the linear slope on the DS disc should be within 1 sigma of 0. The result, for negative lunar phases, is this:
Here is a sharper pdf:
In the upper frame above, we plot the B-V difference between the BS and the DS. On average, B-V seems to be 0.4 bluer on the DS than on the BS. Given that the BS has B-V near 0.92 [Allen, 3rd ed.] we can convert the above to B-V for the DS by subtracting it from 0.92. That would give us B-V values for the DS near 0.5. Fred Franklin found less blue values since his B-V for DS was 0.17 below the BS value. That is at the lower limit of what we find.
We plot +/-1 error bars - they seem to get bigger as phase approaches Full Moon. We note that the scatter is larger than the errors - so something other than the errors we can estimate are causing the scatter.
Is there a trend in the above? By eye, it seems like there is - as phase approaches FM we have more positive ordinates - implying that the DS gets bluer with lunar phase.
The points fall in 'columns' corresponding to single nights - so something is causing a colour-change as the night progresses. All data are of course extinction-corrected so we doubt that the nightly colour-change is due to extinction. Is it a true color-change in earthshine?
We can compare the photometric B-V derived in the above way with B-V differences in albedo found by fitting the DS disc edge.
Now for the lower panel:
In the lower frame we plot B-V against the PSF-parameter alfa needed to bring the sharper image into the same state of blurriness as the less sharp image - so that halo cancellation could occur. There is a very clear relationship between alfa and the B-V achieved. I guess this is bad news? Nature (i.e. the B-V colour of Earth) should not depend on a PSF parameter used in the above convoluted way. Oh dear.
I still think it is worth comparing the present results with the ones based on fitting the DS edge. If we get a contradiction it may mean anything - but a strong agreement would imply that the 'edge-fitting method' has some gremlins hiding in it - i.e. that results are very dependent on those alfas.
It is also possibly revealing that the nightly points fall in columns [probably sequences once telescoped out to single nights - will check]. It is telling us something about things that change on an hourly basis - this includes airmass (i.e. extinction) and also the Earth-Moon-Observer aspect - so, still interesting!
Real World ProblemsPosted by Peter Thejll Jun 12, 2013 07:30PM With the whole telescope system back at DMI we are working to bring it back to life. After that we will try to fix the problems that beset us.
So far (latest activity on top) we have this:
July 8 2013: All PCs on internet reachable from outside, and so is the iboot and the NAS c an be seen from various machines.
July 4 2013: ibootbar is now reachable from woof by using, in browser on woof, 'ibootbar'. On egregious you need 'ibootbar.usr.local'. Eigil says woof is reachable from home using woof instead of via nordlyset in ssh menu. egregious should also be thusly reachable from ssh.
July 2 2013: Eigil Pedersen has made it possible to reach the woof linux machine by giving it an IP number. It can in turn see the iBoot viaits 192 IP number. Working on the NAS.
June 24 2013: Located the Aurora cloud sensor, and downloaded software. Will try to set up on Wine/Linux machine.
June 20 2013: Working to install static IP-numbers. Connected DEC-axis motor and its limit-switch so that we do not have to rely on dubious short. Devices still do not react when acessed from LabView.
18 June 2013: We figured out that the limit switches must be connected - then the mount comes alive. LabView still freezes when you click on whatever.
June 14 2013: Wired up part of the mount head with the Dome Breakout Box. Still need to understand why LabView is talking about "no licenses" for "Report Generation Toolkit" and the "Internet Toolkit". Where were the licenses before?
June 12 2013: Booted up the Watchdog, woof and PXI computers. Labview looks strange - some VIs are not working and then none will run?
June 11 2013: Reset all the 115V selectors to 230 V and applied power - nothing blew up!
May/June 2013: Uncrated the control rack and the telescope tube. Inventoried the other boxes.
Real World ProblemsPosted by Peter Thejll May 05, 2013 02:31PM Our efforts to measure albedo precisely and accurately are limited by the natural variability. Albedo is inherently a quite variable property of the Earth - viewed day-to-day [see presentation linked to below for details on the GERB data variability!]. Just how much does global-mean albedo vary - in the short run and over many years?
We have CERES albedo data for the period 2000 to 2012. It is given as monthly mean data for a 360x180 degree grid. We take this and calculate global-mean monthly values. Some results are here:
Data downloaded from: http://ceres.larc.nasa.gov/cmip5_data.php these are the monthly-mean data prepared by NASA for the CMIP5 effort.
I calculate the global weighted means for the given monthly-mean values of upward and incoming shortwave light at TOA.
The linfit slope is:
Slope: 5.4920926e-08 +/- 5.7758642e-07 in albedo units/day.
This is an INsignificant slope - per decade it would amount to 0.09 % of the mean albedo. Using, instead of the slope, the +/- errors on the slope we get: +/- 0.97 % per decade.
The mean we get is:
Mean albedo: 0.218214
this is small. Publications using these data say: 0.29 ... there is some problem with accounting for regions' weights. I omit all areas where the incoming flux at TOA is less than 12 W/m² - this helps avoid Inf's and NaN's. I weight each area with the cosine of the latitude of cell middle.
We learn that albedo is remarkably constant when observed by satellite. There is no discernible slope to the data but if we use the 1 sigma uncertainties on the slope as upper limits we find that per decade the albedo has changed less than 1%.
The monthly means follow an understandable annual cycle (maxima in NH and SH winters with minima in March and September). The spread around monthly means amount to 0.25 to 0.6% of the monthly mean value.
Climatologically it is an open question whether albedo ought to change with climate drift. During the observing period global mean surface temperatures have changed by about +0.1 degree C [see http://www.ncdc.noaa.gov/sotc/service/global/global-land-ocean-mntp-anom/201101-201112.png ]. This is during the 'hesitation period' that is much discussed presently. During other decades mean T has risen much more - but we have no albedo data from these periods.
Using the above 1 sigma upper limits on slope of 1% per decade we see that if the slope is due to changes in T then the relationship is 1% per 0.1 degree or 10% per degree. This is based on an upper limit and the true value is closer to 0% per degree.
Note that the previous argument is unrelated to the EBM based relationship that works for equilibrium climate only - there, and only there, the expected relationship is -1% per degree.
So, what does that give us? If we were to observe a larger slope we could use the data in the "satellites are getting it wrong" mode - as Pallé et al did for a while. If we measure no slope we can hope to set more stringent limits to the slope than the above satellite values do - can we determine the slope of global mean albedo to better than the 1% per decade above? In this presentation I found upper limits of 0.2% per decade based on a null hypothesis of 'no albedo change' and realistic observing limitations. The numbers used were based on Frida Benders CERES data, but they do not differ enormously from the present, longer, ones.
So can we reach 0.2% error per decade, observationally?
This requires a discussion of the single-frame errors we get as well as the period-mean data we can expect. More alter!
Data reduction issuesPosted by Peter Thejll Apr 17, 2013 04:52PM We are using co-add mode - that is, we take stacks of 100 images and co-add them to increase the SNR. Since there is RON (read Out Noise) in each frame - about 2 ADU/pixel, tests have shown - it is important to know whether a sequence of added frames have a mean that converges.
We now test this directly on the stack of 100 images: "2456015.7742682MOON_V_AIR". We extract a small square of size (2*w+1) around regularly spaced points covering the image plane and calculate a mean subframe (averaging along the stack direction), the average value of that mean subframe (a scalar for each of n coadded subframes), and plot these. These are shown in the PDF file:
The top panel in each page shows where the point was chosen. w was 4 so subframes are 9 pixels by 9 pixels, centred on the point. Second panel in each page shows how the mean value of the average subframes evolves, along with estimates of the +/-1 sigma error bounds (calculated based on the value the series converges to, or at least evolves to, in the last 10 steps of the series; the known RON (about 2.4 ADU/pixel in all observations, estimated from bias frames differences), the number of pixels in each subframe and the number of subframes co-added). Last panel shows the mean value of each sub-frame used (along the stack direction).
What do we see? In page 1 a point on the sky has been selected and we see a mean value that evolves inside the expected error bounds, and we see that the subframes have mean values with some spread, but no trend.
Page 11 shows a point on the DS. Nice evolution.
Page 24 is on the BS and is very hard to explain. This holds also for points 'near the BS'.
It seems we have little to worry about on the sky and on the DS, but on the BS we see strange evolution of the running mean! We have before touched on such subjects in this blog, when we tested effects of alignment of images - perhaps this problem was hiding inside the other problem?
[added later:] Chris asked a good question. Here is a plot of the running mean of the whole frame, as frames are taken from the stack. The value is expressed as a percentage deviation from the middle value on the curve - about 1378 counts. A small drift is seen at the 0.05% level from one frame at left to all frames at right. I think this could be due to sky variations - or slight drift of the Moon inside the frame causing 'light to fall off the edge'. If drift is the explanation we may have an answer for why the mean in a much smaller sub-frame, when near the BS, drifts so much more - image brightness gradients are being sampled in a small subframe. Experiment at top should be redone but with drifts taken out. No large drift was evident sp I guess we are learning that a small drift can be very important!
This has an implication for how we measure the albedo - if we use 'DS patch divided by BS patch' we run a risk . if we use 'DS patch divided by total flux' we are much better off.
From flux to AlbedoPosted by Peter Thejll Mar 29, 2013 03:21PM [edited versions:] We have discussed many ways of extracting albedo from our data. We first considered the 'ratio over ratio' methods - they consisted of extracting counts from patches on areas of the scattered-light cleaned images in the DS and the BS - the ratio of DS/BS observed to DS/BS model is proportional to the albedo of the Earth. The other method, used more recently, is based on 'profile fitting' near the DS sky edge.
We have now arrived at comparing the 5-color albedos derived from these two methods:
Right: Albedos for positive and negative lunar phases (Full Moon is at 0 degrees) from the DS/BS method where the "BBSO log" method has been used to remove scattered light. Left: Same but for the profile fitting method. Note different
vertical scales on axis. The same nights were considered for both
plots, but not all are present in both, due to outlier removal, etc.
We see, for both methods, a rise in the derived albedo as phases nearer Full Moon are considered. This is possibly due to effects of scattered light from the BS which has been incompletely handled by the respective methods. The values found with the two methods are quite similar - apart from increased scatter and less colour-separation in the DS/BS method. The data for the positive and negative branch of the phase diagram are not similar, in either case.
From tests shown elsewhere in this blog we do not expect the halo to interfere with the DS for large absolute lunar phases - i.e. near new Moon. The above diagrams shows lowest values for phases near 110 degrees. There is a slight increase in values larger than this - what can that be due to, if the halo is interfering less and less? Well, we must again remember that the above results are model-dependent and the Model may be adding its own fingerprint. For instance, it may be that the synthetic images we use have a phase-dependent error in their representation of lunar and/or terrestrial albedos. Note that the same synthetic models are used in the DS/BS method as in the profile-fitting method. This question can be addressed by studies of the effect on the 'bend' seen above of different BDRF models.
Bias and Flat fieldsPosted by Peter Thejll Mar 23, 2013 06:24PM We show here the B-V mean value of all pixels lit by sunshine. The data have been selected for being 'good', so the scatter is somewhat disappointing. Best one can say is that we do not contradict the published B-V=0 .92 value.
Blue and red points correspond to different parts of the lunar cycle - before and after Full Moon - waxing and waning, whatever you want to call it.
Data reduction issuesPosted by Peter Thejll Mar 08, 2013 09:25PM During reduction of our data it has become evident that the VE2 filter gives us certain problems. A new problem is that VE2 exposures tend to have a 'sky component' that is much larger than in the B,V,VE1 or IRCUT filters. Here is a plot of the relationship between exposure time and the 'pedestal' - i.e. the offset that has to be subtracted from the VE2 image to bring the sky level down near to 0 (in bias-subtracted images, of course).
It does seem like the pedestal height is a decreasing function of the exposure time. Why is it any function of exposure time, when the other filters do not show this behaviour?
If there was a source of light in he image frame, then its level should INcrease with exposure time, surely? Here is the plot for the VE1 pedestal:
The level is much smaller than in the VE2 case - and it does not appreciably depend on exposure time.
From flux to AlbedoPosted by Peter Thejll Feb 23, 2013 06:53AM The best data will be those that have almost no interference from the halo at all. Those exposures, if they can be identified, will have the least scattered light on the DS and will therefore be the easiest to clean up. Here we try to identify those exposures. We simulate the effect of the halo on ideal images and measure the amount of flux added by the halo - before any attempts to remove it. We loop over relevant values of lunar phase and 'alfa', the PSF-width parameter. We calculate the change in flux for a region near the DS rim and express the change in percent. The contour plot shows contours at 1,2,4 etc percent change. Overplotted are red data points showing locations of real exposures.
We see that the least polluted images are found at large phase (near New Moon) and for narrow PSFs (large alfa values). This is not surprising! We see that we have several exposures taken under conditions that allow just 1,2,4 etc percent change in DS intensity.
If these images are the least polluted by BS-scattered light, we should have the most success when applying our scattered-light removal techniques to these. If they have a 10% effect due to scattered light, and we can remove 99% of that we have reduced the error induced by the BS halo to the 0.1% level.
Here are albedos determined for 6 nights. We see 4 filter bands (VE2 is still being processed). Each color represents one night. There are 3 determinations of albedo from each image - hence points come in columns of three.
1) Albedos can differ in level from night to night - e.g. blue and red in panel 1.
2) Albedos can also be almost constant during a night - e.g. purple and green points in panel 1 vs red or blue.
3) Some nights have large scatter - e.g. green vs blue
4) B albedos are higher than V and others
5) VE1 and IRCUT albedos are very similar - the filters are also very similar, so this is good for reproducibility.
6) While some albedos evolve smoothly on a given night IRCUT shows a 'dip' - e.g. blue crosses in B vs blue points in IRCUT
Questions of interest:
a) Why are some nights noisy? E.g. the green points. Information on airmass, alfa and bias frame statistics, and RON, are available.
b) Can the nightly evolution (i.e. slope) of the points be related to anything happening on Earth?
c) Can the change in slope in IRCUT be related to anything special?
Now, we consider the variance-over-mean along the stack direction - that is, for each image pixel we calculate variance and the mean and look at the ratio of these, which we call 'VOM'. In the perfect world, this image should be 1.
We look at unaligned images, and then at aligned images. The 'superbias' was subtracted (but not scaled to the adjacent dark-frame average). RON of 8.3 electrons was subtracted before VOM was calculated as (SD-RON)^2/mean. Gain used was 3.8 e/ADU. We plot VOM in a slice across the final images:
Top is VOM in the unaligned image, and below is VOM in the aligned image. A lot can be mentioned: First, the surface is not constant, obviously. Second, the effect of alignment is not just a uniform lowering of the VOM that we expect (same mean, less cross-talk between pixels).
In the top image we have VOM near 0.1 (dotted line) on the DS and most of the sky. On the BS the VOM is near 10 apart from the peaks that occur at the intersection with the lunar disc edge. There variance rises because of jitter between images and the mixing of disc and sky pixels.
In the aligned image VOM is near 2 or 3 on the BS disc, higher at the peaks (so there is still an edge-effect, but less). On the DS and the sky a spatial structure has been formed, slanting inwards towards the BS.
What is going on? Effects of pixel-subshifting? What does the interpolation do? Why is it spatially dependent? Strange flux-nonconservation?
The sub-pixel shifting used is based on the IDL 'interpolate' function. In the manual for this function it is suggested to allow for cubic interpolation by setting a keyword "cubic=-0.5". I did this and recalculated the above. The effect was slight.
A test was performed to see what variance is induced by the interpolation. By shifting an image by a non-integer amount of pixels in x and y and then shifting that image back, and calculating the difference, we see that using INTERPOLATE without 'cubic' induces 2-3 times more residual S.D. than does use of INTERPOLATE with cubic=-0.5. The interpolations lost variance compared to the original image. With cubic interpolation the loss of standard deviation relative to the mean is at the 1% level - quite a lot actually. [Note: conservation of flux not the same as conservation of variance]
Could it be some effect of bias-subtraction? VOM has dropped most in the halo near the BS. Why?
Control SoftwarePosted by Peter Thejll Feb 09, 2013 03:19PM In a potential upgrade of what we currently have, we might consider going to linux and freeware, for our command system.
Some devices can be operated via the indiserver concept. Devices for which drivers have been written can be commanded from essentially the linux command line. This means that scripting systems are easier to set up - for us at least.
I have been able to control our SXV-H9 CCD camera using this system. Here is a link to how it is done:
There exist indiserver drivers for gphoto2 - that means that Canon DSLR cameras, for instance, can be operated in this way.
A TrueTech filterwheel is being tested. LX200 telescopes can be controlled - i.e. mounts such as ours.
Data reduction issuesPosted by Peter Thejll Jan 26, 2013 05:29PM In this post: http://iloapp.thejll.com/blog/earthshine?Home&post=293 I proposed to fit more elaborate PSFs than we have been using.
I present here the results of fitting a 2-parameter PSF. The previous one was characterized by an exponent alfa, and had the form
The new one allows two exponents - one regulating the slope of the PSF up to a cutioff point in terms of radius, and the other parameter taking over outside that point. Thus we have 2 new parameters to fit - exponent and cutoff point.
Using such a PSF and evaluating success on the parts of the image designated by the mask - and using a mask consisting of sky on both sides of the Moon, but no polar areas - i.e. a 'belt across the image' - we arrive at images with slight changes in the estimated halo on the DS of the Moon and larger changes in the area on teh BS-sky.
This plot summarizes what was found:
The top panel shows absolute value of residuals (obs-EFM-model) in a slice across an image after the halo has been removed (along with most of the BS, as is always the case in the EFM method). The PSF used was the new tow-parameter PSF. The middle panel shows the result when the usual one-parameter PSF has been used. The last panel shows the difference between the two panels, expressed as a percentage of the first one. The illegible y-axis label for the bottom panel shows 10^-2 at the bottom, next label is at 10^0.
Note that absolute values of residuals are plotted above.
We see the DS to the left, the almost-erased BS to the right and the sky on either sides. Clearly, the BS sky has been removed much better in the first image - i.e. with the two-parameter PSF. On the DS we see changes in the range from 1 to a few percent on the sky-ward part, rising to tens of percent near the BS.
This implies that a signifcantly different amount of scattered light has been removed using the two PSFs - but which is the better result? Judging by the RMSE per pixel on the mask in the two cases there has been a significant improvement in going from one to two parameter PSFs. The RMSE per pixel in the one-parameter PSF case is about 5 [counts/pixel], while the RMSE per pixel is 0.23 [counts/pixel] in the two-parameter case. Most of this change has evidently taken place in the BS part of the sky.
We need to know if there has been an improvement on the part of the image that matters - the DS!
The best fitted parameters were quite alike - near alfa=1.7. The best-fit cutoff point was near 30 pixels.
Exploring the PSFPosted by Peter Thejll Jan 26, 2013 10:45AM It seems that success for us will be linked to our ability to correctly remove scattered light from the BS. This hinges on our understanding of the scattering model - in essence, the PSF.
Currently we use a PSF that is empirical. The core is made up of values from a table generated from observations of bright stars and Jupiter. To that we link an extension, also empirical, that is based on what the far wings of the lunar halo looks like. This PSF is then 'exponentiated' during fitting to actual images in the EFM method.
We noted during several posts below that the values for the exponent varied little and only now and then seemed linked to the extinction. This could be an indication that most of the time (during good nights) we are limited by something fixed - such as the optics - rather than atmospheric conditions. On the other hand it could mean that the PSF is not very accurate in its basic from and that the fitting method gives up at some stage, leaving us with an exponent that is somewhat random, and therefore not linked to the atmospheric conditions.
We should also recall what happens during application of the EFM method: We have tested various forms of sky masks for this method - some that allowed fitting emphasis on both the DS and BS sky, and others that emphasized only one side. Common for the ones that focused on either just the DS sky or the BS sky was that the halo shape on the other side was not very good.
Might these things be indicating that a better PSF should be generated or a better way found to apply the fits?
I'd like to suggest the following: Perhaps the PSF has a form like
instead of the present
PSF ~1/R^const_alfa ?
I would like to try to use a piece-wise constant alfa so that the PSF is separated into radial zones, each of which has its own alfa, found by fitting.
Optical designPosted by Peter Thejll Jan 26, 2013 10:06AM In this post: http://iloapp.thejll.com/blog/earthshine?Home&post=289 the importance of the SKE for scattered light was discussed. The images shown, though, were not shown fairly - with intensities scaled to comparable levels. I therefore extracted a line across the BBSO image and a line across our image, at right angles to the SKEs, rescaled the intensities, aligned the plots and get this: The black curve is from our image (whichis a sum of 10 well-exposed images). The red curve shows the cut across a single BBSO image.The BBSO image has only the DS peeking our behind the SKE, while our image has the BS in full view.
We see entirely comparable 'halos'! The BBSO image ha a more pronounced sloping 'tail' onto the black side of the SKE than we do, and more noise. If that sloping tail is 'halo' we had a better system than the BBSO!
What does the above mean? It does NOT mean that we have less 'halo problems' than BBSO does - because the BBSO expose their DS so that the halo from the BS is not allowed to be formed. Yes there is a similar halo from the DS on their images as there is from the BS on our images - but the halo from our BS is very much stronger than their DS halo.
When the BS halo is small - i.e. near New Moon - we have minimal effect of the BS halo.
From flux to AlbedoPosted by Peter Thejll Jan 25, 2013 04:02PM We have now reduced all the good data, and have arrived at the intrinsic properties of each image. We extract now the ratio of the DS-patch to the total flux and divide this by the similar ratio extracted from the synthetic model. This ratio of ratios is the same as the 'Lambertian sphere terrestrial albedo' and is the quantity published by e.g. Goode et al.
We select data for airmasses less than 2 and for statistical error less than 1.5%. This error is that due to mainly pixel-selection inside the 'patches' on the lunar surface (one near Grimaldi, one inside Crisium). Monte Carlo bootstrap sampling of the pixels inside a lon-lat box were resampled, with replacement, and the standard deviation of the consecutive means calculated. Errors due to image alignment, improper scattered-light removal, synthetic model problems and so on, are not included.
Here is the plot of extracted albedo against lunar phase for the VE1 filter (the filter with least scatter: IRCUT is almost identical to this plot): [scroll down for discussion]
The colours designate the method used for scattered light removal: red is the EFM method, orange or yellow is the 'linear BBSO method', and green are the raw data. Crosses and diamonds indicate the sign of the lunar phase - crosses are for positive phase. Error bars (error due to counting noise, not image alignment etc) have been plotted over the symbols, but are all smaller than the symbols themselves.
What do we see? There is scatter and there is dependence on lunar phase.
First, let us discuss the lunar phase dependency: For small phases we are essentially getting closer to Full Moon and the scattered light is more and more of a problem. The Raw data are doubled by phase 80 degrees, showing the importance of removing the scattered light. BBSO-linear is better than no removal all while EFM is better than BBSO-linear. There is also another lunar phase dependency - notice how the albedo rises for large lunar phases. Since the halo is smallest here this is not an effect of scattered light! The albedo is a composite quantity, consisting of quantities measured in observed and modelled images. The scattered light influences mainly the model, and mainly at small lunar phases. The reflectance model soley influences the synthetic lunar image, of course - I think we are seeing the influence of the (inadequate) reflectance model at large lunar phases - the albedo rises artificially due to this.
The there is the scatter. We see 'daily progressions' for some connected lines of data points. For instance, by 90 and 95 degrees phase. These points represent our candidates for geophysics! I think the 'progression' along these lines are due to different parts of Earth )oceans, clouds, continents) rotating into view for these observing sequences.
Below is the same plot, for the B filter. Apart from more scatter we see that the EFM seems able to remove most effects of scattered light, compared to the BBSO-linear method. A stronger 'upturn' is possibly seen at large phase.
Data reduction issuesPosted by Peter Thejll Jan 23, 2013 03:24PM Chris thoroughly investigated the quality of all data and arrived ta a list of the 534 best images: http://iloapp.thejll.com/blog/earthshine?Home&post=277 I have collected all those images in 'cubes' consisting of raw observation, EFM- and BBSO-cleaned images, images for incidence and emergence angles and the ideal synthetic image and lunar longitude and latitude images - each 'cube' contains centred images for a given 'stack' of 100 observations. We thus have 534 'good coadded' images with all secondary data that belongs to that moment. Each cube is about 16 Mbyte uncompress,a nd slims down to 5-7 Mbyte upon being bzip2-ed.
I inspected the correlation of these observed stacked images with their synthetic models and found that 8 of them are poorly correlated with their synthetic models. I inspected each case and found that the problem was either focus (the observed image is blurred) or a slight alignment problem.
Omitting those images we arrive at this list of 525 best images:
These are all available as 'cubes', upon request.
In the future we might wish to only work with these images, since alignment and image-reversal issues have been dealt with (the observed images swap East and West upon the meridian flip).
We can set up a Mark I dataset, to be updated later as we find issues or solve the remaning (small) alignment issues.
We now extend the method to include an identical analysis step applied to synthetic images generated for the moment of observations. We then take the ratio of the results from observations and the results from the models. This will eliminate the effects due to geometry and reflectance (as long as the reflectance model used for the synthetic images is correct), leaving only the effects of changes in earthshine intensity. So, this is another 'ratio of ratios' result. The ratios involved are, in summary:
(Laplacian signature at DS/ Laplacian signature at BS; in observations) divided by (Laplacian signature at DS/ Laplacian signature at BS; in models).
We use only the 'good images' identified by Chris in this entry: http://iloapp.thejll.com/blog/earthshine?Home&post=277
We start by inspecting the dependency of (Laplacian signature at DS/ Laplacian signature at BS; in observations) on phase [attention: this is not yet the promised ratio of ratios - just a ratio! :)], in each of the available filters:
We see the morning and evening branches folded to the same side, for comparison. We see an offset between the branches in each filter, and we see the dependency on lunar phase. We next look at the same ratio but from models. Since we do not have colour-information in our models we redundantly now show plots for each filter as if the models were color-dependent - they are not: but the points available in each filter are different, of course. Recall that a model is generated for each observation:
We see a very similar pattern - dependency on phase (but steeper this time). We also notice that the branches are closer together than in the observations.
What does this mean? The branches are separated, in observations, due to different distributions of light and dark areas on the eastern and western halves of the lunar disc facing Earth. Our Laplacian method samples pixels right at the edge of the lunar disc and has evidently met areas of different albedo. The separation in observations is not reproduced in the models - this can imply that the model albedos are incorrectly distributed (i.e. mares and craters etc in the wrong places) this is somewhat unlikely as we use one of the most detailed lunar albedo maps, from the Clementine mission. However, our model does not sample colour - the map used, and thus the ratio between 'light' and 'dark' - is taken from the 750nm Clementine image [I think; must check!]. This map was stretched to match the older Wildey albedo map [see here: http://iloapp.thejll.com/blog/earthshine?Home&post=253 ] which was made in such a way that the 'filter' the WIldey map corresponds to is a combination [see: http://iloapp.thejll.com/blog/earthshine?Home&post=286 ] of the Johnson B and V filters. We see the most well-reproduced branch spacing in the V band observations, compared to the models. This implies we have some colour-information about the two halves of the Moon - or a tool for how to scale the lunar albedo map when different colours are to be considered.
We next inspect the ratio of the observations and the models - the ratio of ratios:
For each filter is shown the Albedo derived - it is the 'ratio of ratios' spoken of above and is identical to the one used in BBSO literature - it is the albedo relative to a Lambertian Earth-sized sphere. The model albedo used was 0.31 so the 'actual' albedo derived is the above times 0.31.
We see that there is a phase-dependence in this - particularly in one branch, with the observations being relatively brighter than the models at phases nearer Full Moon, compared to phases nearer new Moon.
Since we have seen the EFM method produce less phase-dependent albedos [see here: http://iloapp.thejll.com/blog/earthshine?Home&post=252 ] we think the Laplacian method needs further development and investigation before it can be used.
It is worth listing why the Laplacian method might be useful:
1) It does not require careful alignment of model and observation. The signal is extracted from a robustly defined location in each image. 2) It is a 'common-mode-rejecting' method and is not dependent on image resolution. 3) It is relatively fast.
One of the limitations we realized then was that image resolution influenced the results from the Laplacian edge signature - the more the edges of the image are smeared the lower the Laplacian signature estimate will be.
We now revisit the method after realizing that the whole image is affected in the same way by resolution issues - the intensity estimate derived from the DS and BS edges will both depend on image resolution in the same way. The DS estimate is proportional to earthshine intensity, while the BS estimate is proportional to 'moonshine' intensity (plus earthshine - a vanishing contribution). The ratio of the two estimates will be independent of resolution - i.e. the ratio has the 'common mode rejecting' property.
We convolved ideal images of the Moon with the alfa=1.8 PSF and estimated both the 'step size' at the edges (on DS as well as BS) and the Laplacian signature. We plot them below:
The top panel shows the step size estimate against the Laplacian signature estimate on the DS only. We see an offset, with Laplacian estimate being smaller than the step size estimate. This is caused by the resolution issue - the edge is fuzzy and the Laplacian is degraded. The step size is estimated robustly from the original ideal image (i.e. before the convolution is performed) and is thus not dependent on image resolution. The bottom panel shows the ratio of DS and BS estimates - x-axis is the step size estimator and the y-axis is the Laplacian estimator. We now see that the dependence on image resolution is gone from the Laplacian estimate. The correlation between the two estimates is good, at R=0.92, but not perfect. Noise was not added to the images. Estimates of Laplacian were performed on the line through the disc centre. The step size estimator is based on the average of 9 lines through the disc center.
[Added later:] Since we are working with synthetic models we know the actual earthshine intensity in each image, so we can compare what the actual intensity is to what is found with the 'step' and the 'Laplacian signature' methods:
In the first panel we see what we saw above - that the step and Laplacian methods are rather consistent. In the panel below we compare to the actual earthshine intensity.
Since the Laplacian signature method (and the step size method) literally express the DS/BS intensities ratio (which is only proportional to the earthshine intensity) we have different values along the x- and y-axes. The geometric factors having to do with distances and Earth's radius are not compensated for.
The main result is that the relationship between actual earthshine intensity and the Laplacian signature method (and by extension, the step size method) are not quite proportional - there is a slight curve. In interpreting the above we should keep in mind that the Laplacian estimate of DS illumination from the edge derivatives is in a different role than the estimate of BS illumination - the former is not very geometry dependent while the latter is: That is, earthshine is due to light from a source we are sitting on doing our observations from, while the BS illumination is more angle-dependent in that the reflectance properties of thr Moon come into play to a larger degree - at the moment I am not sure whether there is an angle-dependence 'along the BS edge of the disc' that can cause a problem in interpretation. Will need to look at this.
Another thought: is it possible to make a calibration relation between actual earthshine intensity and the Laplacian DS/BS signature estimate?
The first images we look at are not the bias-reduced images Chris uses, but 'EFM-cleaned' images (i.e. scattered light has been removed). Both the B and V image used were co-additions of 100 images each. The two images were centered in the image frame. We look at tow sets of B and V images - the one at the top is cleaned with one setting of the EFM method; and the one below is done with another setting of the EFM method. Some of the features to the left on the sky are the effect of 'cyclical overlap' from the right side when the image is shifted.
We see the DS to the left and the BS to the right. The BS halo and the BS have become undefined (i.e. 'NaN') because either B or V is negative here (remember, it is not a bias-reduced image with full halo in place - these are EFM-reduced images so the BS and parts of the halo are now zero or negative. In overlap almost all of the BS and the BS halo have become NaNs!
Here is a 'slice' across the middle of the upper image:
And here is the same slice across the lower image: And finally, here is the slice across the un-reduced B-V image - that is, the image formed from B and V images calculated from the images that were only boas-reduced (as in Chris' plot). The uppermost image has a large 'dip' when we get near the BS and its halo remnant . In the middle image the halo has apparently been better removed. STrangely, thereis least sign of a halo in the 'raw' images where nothing has been done to remove the ahlo. This needs to be discussed!
We used other airmasses than Chris. In my calculations the airmasses for the two images involved are:
B_am=2.545 image is 2456034.1142920MOON_B_AIR_DCR.fits V_am=2.477 image is 2456034.1164417MOON_V_AIR_DCR.fits
I used kB=0.15 and kV=0.10, like Chris.
I did not solve for B and V by solving two equations with two unknowns - I iterated. Convergence was fast. I iterated on the whole images.
The first of the above EFM cleanups was done with a weighting of the mask used to define the area of the sky on which to reduce the sum of squares that favored the DS part of the sky. In the second attempt above equal weight was given to the RH and LH sides.
Data reduction issuesPosted by Peter Thejll Nov 04, 2012 01:29PM While waiting for the CCD to be fixed, I am summarizing the discussions we have had on this blog. The material should be used largely in section 6 of the manuscript we submitted, leaving the other parts of that paper - or splitting it into tywo papers. Here is the submitted manuscript as well as the start of a summary the summary is really just a collection of links to this blog.
Real World ProblemsPosted by Peter Thejll Oct 19, 2012 10:58AM The Moon is not observable all the time from a single observatory, all year round. To get complete - or almost-complete coverage we ask: where should the earthshine telescopes in the network be placed? The plot below shows, with coloured symbols, when the Moon is observable (defined as Moon up and Sun down), for 5 observatories around the world. MLO is Mauna Loa, MSO is Mount Stromlo in Australia, LCO is Las Campanas in Chile, SAAO is South African Astronomical Observatory. The underlying curve shows the earthshine intensity for one month. The Earth was modelled as a cloud-free sphere with bright continents and dark oceans - hence the zig-zag nature of the curve. Addition of randomly placed clouds would tend to dampen the amplitude of the zig-zags by about a factor of two.
When the underlying curve is not covered by coloured symbols it means that the Moon is not observable from any observatory. We note that this happens particularly when the earthshine is intense - that is near Full Earth, which is New Moon. This is clearly because the Moon is almost never in the sky alone when it is New (it has either risen shortly before the Sun, or will set soon after the Sun). Observability is best near Full Moon (New Earth) near day 15 - but then observing earthshine accurately is very difficult.
On average a single observatory experiences the Moon above the horizon and the Sun below, 25% of the time. Given the choice of these 5 observatories the total observability depends on time of year - given in the panels.
Clouds and the need for the Moon higher in the sky cuts the observability.
The above simulation is for two months of the year - the declination of the Moon changes with the seasons so the relative contribution by each observatory changes with time.
Residuals image of Moon minus model of BS and its halo, fitted to the sky (i.e. the EFM method). The green and blue colors are a few tenths on both sides of 0. The mask used covers the image except the disk of the Moon.
In this application of the EFM to an observed image of high quality we see that the residual sky is not even. Particularly on the BS (to the right) there are some streaks. A blob at the bottom also shows unevenness, as does the striations to the left of the DS.
This illustrates a shortcoming of the EFM. Since the PSF used is rotationally symmetric it is possible that the unevenness seen above are in the observed image - perhaps some sort of reflection effects in the optics. The EFM is a global method seeking to subtract all the scattered light with one model. The BBSO method is essentially local - it estimates the scattered light in a wedge off the disk. On the other hand the BBSO linear method almost certainly removes too little of the halo as you get nearer to the BS. The EFM method attempts to model all parts of the halo.
Control SoftwarePosted by Peter Thejll Aug 25, 2012 08:50PM Following the power surge and the consequent failure of the system, I installed SOLIS in case the software and drivers had been hurt by the power surge. This was apparently not a good idea - during construction of the system it was realized that 'drivers' from SOLIS and 'drivers' from the SDK are not the same and that only drivers from the latter must be present.
I therefore today de-installed (using the Windows 7 Control Panel) the Andor SOLIS package - as well as the unwisely installed Andor SDK package (both, apparently, with their provided drivers).
I then tried to run the 'setup.exe' present in the Andor SDK zip file - the setup offered to install itself into
C:\Program Files\ANDOR SOLIS\Drivers
which is supposedly not right - the SDK should install into
Error budgetPosted by Peter Thejll Aug 21, 2012 12:33PM A required image analysis step is the determination of lunar disc centre and radius.
We currently use a hybrid method: First we fit circles to points on the BS rim (found as an image by using edge-detection image-analysis methods: SOBEL filters!) of the lunar disc image - this is done many times using different points and then the median is extracted for x0,y0 and radius. These values are then used as starting guesses for a more refined method that searches a range of possible values near the starting guesses and determines a 'winner' based on how well a synthetic circle matches the 'rim image' generated above.
We compare the statistics of the starting guesses and the final adopted values. In two separate runs on 88 different images we assemble these values:
Run 1: x0 start guess: 311.82352 +/- 2.7447506 x0 final guess: 310.44694 +/- 2.3127377 y0 start guess: 248.49938 +/- 1.1339389 y0 final guess: 248.70556 +/- 1.0812740 radius start guess: 136.32425 +/- 0.98834739 radius final guess: 136.90833 +/- 0.44461002
Run 2: x0 start guess: 311.78076 +/- 2.9802184 x0 final guess: 310.46767 +/- 2.3079720 y0 start guess: 248.51968 +/- 1.0747726 y0 final guess: 248.70327 +/- 1.1017349 radius start guess: 136.42162 +/- 1.2634192 radius final guess: 136.90870 +/- 0.40265642
x0 is found slightly more precisely with the final method (2.3 pixels vs 2.8). y0 is found with similar precision in the two methods (about 1.1 pixels). radius is found better with the final method (0.4 vs about 1 pixels).
We seem to be able to determine the centre of the lunar disc with image analysis techniques to the 1-2 pixel level, and radius to better than half a pixel.