Earthshine blog

Earthshine blog

"Earthshine blog"

A blog about a system to determine terrestrial albedo by earthshine observations. Feasible thanks to sheer determination.

Stdev and bias in EFM

Error budgetPosted by Henriette Schwarz Mar 16, 2012 03:35PM
I have worked with the EFM method for removing scattered light, testing it on a synthetic moon image from night JD 2455923.

The "true" scattered light is represented by an image:
S_true = observed - pedestal - usethisidealimage
The best guess scattered light is represented by the end result from the fit minus the pedestal:
S_efm = trialim - pedestal

I have investigated the mean value of S in the 3 boxes shown below for 100 S_true and corresponding S_efm images. It is the same synthetic image that is the basis of all the "observed" images, but they have slightly different noise added when they are convolved with the PSF.The results:
It is clear that we have significant bias. Not all the stray light is removed with the EFM method. On the positive side, the standard deviations of the fits are small. The counts here can be compared to a typical DS signal of 5 counts.

  • Comments(4)//earthshine.thejll.com/#post171

comparison of alignment procedures

Error budgetPosted by Henriette Schwarz Mar 08, 2012 06:30PM
I have compared two different methods for aligning images: Chae's IDL procedure and Kalle's / my center of mass Python procedure.

The example given here is the result for a stack of 11 images obtained within 0.7 minutes on JD2455864 in the V-filter, and it is representative for many other similar stacks.

I have aligned the stack with each of the two methods, determined the standard deviation of the stack and divided this with the coadded image of the stack. Finally the image has been multiplied with 100, so that it directly shows the relative error on the raw observation in percent. The images are basically the dO/O in the error budget equations.

The left images are aligned with Chae's method and the right images are aligned with the center of mass method. The only difference between top and bottom images is the scaling. The top ones are shown on a square root squale that emphasizes the bright side of the Moon, and the bottom images are shown on a histogram equalized scale that emphasizes the dark side of the Moon.

The center of mass method out-performs Chae's method for both the bright and the dark side. This is especially true for the earthshine. Both methods have difficulties with the bright limb.

The difference between the two methods is not so much the method for calculating the required offset. Instead it is the interpolation technique used in the subpixel move that matters. Chae's method (shift_sub) uses bilinear interpolation, whereas the CM method uses exponential functions as the interpolating functions. With the CM method there is visible blurring of the lunar features, and it is this smoothing that lowers the standard deviation of the individual pixels.

In the form the methods have now, I would recommend Chae's alignment whenever we wish to show a nicely centered pretty Moon image and the CM method whenever we wish to determine the intensity of a box of pixels on the Moon. Ofcourse we can play with the parameters of both methods and this might increase or decrease the blurring.

  • Comments(2)//earthshine.thejll.com/#post165

Coordinates of bad pixels

Bias and Flat fieldsPosted by Henriette Schwarz Jan 08, 2012 12:24PM


  • Comments(0)//earthshine.thejll.com/#post134

Bad Pixels

Bias and Flat fieldsPosted by Henriette Schwarz Jan 06, 2012 09:59PM
Our CCD has a number of bad pixels with lower than average sensitivity. They are a result of imperfect manufactoring, and most of them are collected in small groups. Their numbers and location are stable, and almost all of them are completely removed with even a mediocre flatfield.

But the most severe of the bad pixels, seem to have different strength in different master flatfields, and therefore they show up in diminished form in difference images between two master flatfields. The worst pixels may be 6-7% different from one master to the next, and therefore these pixels should be avoided. Luckily most of the bad pixels are close to the edge of the CCD. There is one semi-severe group in the central part of the CCD. I have seen examples of 1.7% variation in a couple of pixels from this group.

The figure is the twilight master flatfield in the IRCUT- filter from session JD 2455827, shown in power-scale to emphasize the small dark speck that are the bad pixels. The three most severe groups of bad pixels are circled in green, and the two second worst ones are circled in blue. These five groups of pixels should be avoided if possible - just to be on the safe side.

  • Comments(1)//earthshine.thejll.com/#post132

Improved periodic bias plot

Bias and Flat fieldsPosted by Henriette Schwarz Dec 30, 2011 08:21PM
I have improved the plot showing the periodic bias level for multiple areas on the CCD. I have added arbitrary offsets to separate the different datasets from each other, and I have included a small figure showing the location of the five 16x16 areas on the CCD.


  • Comments(1)//earthshine.thejll.com/#post126

Colour filters

Optical designPosted by Henriette Schwarz Dec 27, 2011 02:52PM
The filter transmission as measured by Rodrigo for the B, V, VE1 and VE2 filter up to 800 nm and the IRCUT filter for a much wider range as given by the manufacturer.


  • Comments(2)//earthshine.thejll.com/#post113

S/N image

Error budgetPosted by Henriette Schwarz Dec 07, 2011 04:20PM
From night JD2455864 V-filter moon images I have selected images with similar count levels in the bright part of the Moon. This resulted in 23 images. These I have aligned and coadded (mean-half-median method) thus obtaining a coadded object image O, and a standard deviation image, delO. From these two images (and images with estimates of B, delB, F, delF) I constructed a S/N image calculated pixel by pixel. I found the DS to have S/N ~2-3 and the BS to have S/N ~30-40. The S/N image looks like this

While the DS S/N value might be realistic, the BS S/N value is much lower than expected. This is likely due to the fact that co-add mode necessarily is observed close to a rising or setting moon. On this particular night the Moon was setting, and the 23 selected images are obtained with a 30 minutes interval - in which time the Moon went from an altitude of 22 degrees to 17 degrees. The typical BS count level falls in this period from 27900 to 26400. A decrease of about 5.4%.
Next I tried scaling the images after alignment before coadding. I have scaled them using an 11x11 area in Mare Crisium as reference. This area had a mean in the unscaled coadded image of 10300, and I have scaled so that each image has the value 10500 in that area.
scale_factor = 10500.0/AVG(im[360:371,230:241])
This gave a DS S/N ~1.5 and BS S/N ~100+. The BS S/N seems pretty stable of the exact scaling factor, but the DS S/N increases dramatically (S/N ~12) if I use for instance 15000 instead of 10500. I simply don't think we can count on the S/N values calculated per pixel basis after scaling.

Conclusions and thoughts: If we scale the moon images to counter the setting/rising moon we seem to be able to achieve a S/N on the bright side similar to the poisson noise. It is much more troublesome to estimate the S/N on the dark side. The S/N on the dark side is very sensitive to the method of scaling and the scaling factor. But perhaps a S/N of 2-3 (as determined without scaling using only images selected to have at least somewhat similar bright side count levels) is a decent estimate.

  • Comments(4)//earthshine.thejll.com/#post102

Simple Errorbudget Co-add

Error budgetPosted by Henriette Schwarz Dec 01, 2011 04:06PM
I have aligned and co-added the V-filter moon images from night JD2455864. 16 frames were rejected due to poor correlation when aligning (turned out they were overexposed on the bright limb). This left me with a total of 110 frames.
I have selected an 11x11 pixels area on both the bright and the dark side. The selected areas are shown in the figure below.

The errorbudget formula used is for co-add mode, so exposure time cancels out, and in this first simple version, scattered light is ignored.

For the earthshine I find S/N = 1.7, and for the moonlight I find S/N 128.2.

I have experimented with two different approaches to improve the S/N of the dark side. One way is to increase the number of mean counts, O, in the selected area. For night 864 V-filter O was in the range 398-405. Unfortunately the bright limb was overexposed in the images with higher values than this.
The other approach is to have more images from the same night in the same filter. I have investigated N from 1-1000.
Finding the right exposure time, where the bright limb is well-exposed near saturation, is very important. We can achieve a S/N higher than 3, if we have about 400 frames and a mean dark side count level of 405.
Hopefully we can achieve an even better S/N closer to the new moon. To be continued...

  • Comments(0)//earthshine.thejll.com/#post98

No significant dark current

Bias and Flat fieldsPosted by Henriette Schwarz Nov 20, 2011 02:05PM
I have plotted the mean dark value as function of exposure time from the dark frames obtained night JD2455883. The range of exposure times is 10-200 seconds and thus covers all higher exposure times we might be interested in. No dark current is observed, only the usual scatter due to the 20 minutes period. The period was clearly seen in a plot from the same data of dark count as function of time since first frame.

I think it is safe to say that we no longer have to make dark frames - only bias frames. This will save observing time, and therefore there should be time to ALWAYS obtain a bias frame before and after each science frame or flatfield. This will improve the scaling of the superbias.

The horizontal line in the plot is the mean value of the superbias. It can be seen that it is not in the middle of the scattered dark values. I have seen this before, as well as the opposite with dark values generally being higher than the mean of the supebias. However, in both cases values are within plus/minus 0.5 ADU and the scaling of the superbias means it is not a problem.

  • Comments(1)//earthshine.thejll.com/#post94

JD2455883 darks

Observing logPosted by Henriette Schwarz Nov 18, 2011 04:02PM
I have obtained 100 dark frames with random exposure times in the interval 10-200 sec. This will allow me to determine with certainty if there is any substantial dark current.

The telescope performed as it should and all images were saved without trouble.

  • Comments(0)//earthshine.thejll.com/#post91

Flatfield errors

Error budgetPosted by Henriette Schwarz Nov 02, 2011 06:11PM
I have compared four central 30x30 pixel areas in all the available sky master flatfields. A minimum of 3 masters were available in each filter. The selected regions a (blue), b (red), c (pink) and d (green) are shown in the figure. The flatfield used in the figure is the B master for night JD2455827.

For each filter and region, I have checked the difference between the largest and smallest mean value, expressed as a percentage of the smaller value. These (worst case) changes lie in the range 0.03-0.36%. I seems we can count on the error in a master flatfield to be very low!

The best master flatfields seem to be the B and V filter with only a single case of a change above 0.1%.

The worst region of the four is the green region. This is true for all filters. Perhaps these worst case changes in the worst region can be used as an estimate for the error in a master flatfield?

  • Comments(1)//earthshine.thejll.com/#post78

Comparison of flats within a single twilight session

Error budgetPosted by Henriette Schwarz Oct 28, 2011 06:37PM
I have compared two well-exposed flatfields in the B-filter from the dusk session night JD2455856 with a time difference of about 28.5 minutes.

Each flatfield was bias-subtracted with the proper scaled superbias, had a fitted surface subtracted, and was then normalized. A percent difference image was created as the difference compared to the earlier flatfield.
perc = ((late-early)/early)*100

The mean of this image is of course very close to zero (0.0014), but more interesting is the standard deviation (0.53).

The strongest part of the familiar diagonal pattern is still visible in the percent image (see left part of figure). I have used Image J to rotate the percent image and plot the horizontal profile of the yellow box (top right part of the figure) selected to be perpendicular to the diagonal darker areas. This profile is plotted in the bottom right part of the figure.
It can be seen that the diagonal structure is 0.2pp darker than the immediate neighborhood.

The diagonal structure does change within a relatively short time-frame and this will result in an uncertainty in the science frames when they are flatfield corrected. So far it doesn't seem to be a large change, but more investigations are necessary. The more likely explanation for the change is temperature fluctuations. Perhaps it is possible to investigate if the diagonal structure changes with a period comparable to the change in bias level...

  • Comments(2)//earthshine.thejll.com/#post69

Periodic bias level

Error budgetPosted by Henriette Schwarz Oct 07, 2011 11:50AM

I have ckecked that the periodic behaviour of the bias-level is the same over the whole area of the CCD. I have chosen night JD2455745 for the investigation because all the dark-frames have the same exposure time (60sec).

I found the mean of five small areas of the CCD and plotted them together with the mean value obtained from the full area of the CCD as a function of time since first frame. The selected areas represent the four corners of the CCD (1 pixel away from the edge) and the center of the CCD. I therefore expected that each area would have its own mean level, but the same period and amplitude as the whole frame.

I have tried this for two different area sizes, 8x8 pixels and 16x16 pixels. The periodic behaviour can be seen in both cases, but it is quite muddy for the small area size. The above image shows the 16x16 case. The red stars are the mean values of the whole frame and each colour represent one of the five 16x16 areas. All the five areas follow the same period with roughly the same amplitude as the whole CCD.

From this I conclude that it is safe to assume the periodic behaviour is a shift in the bias-level of the whole CCD area. It is likely to have to do with the thermostatic control of the temperature of the CCD.

  • Comments(2)//earthshine.thejll.com/#post36