[MITgcm-support] Help OpenAD adjfiles
Heriberto Vazquez
heriberto1mx at gmail.com
Tue May 10 18:12:14 EDT 2016
Hello, Thanks for your answer
Yes, you are right I didn't say it properly. gradient check package uses
adxx_files, and for the control variables I am using (u an v components of
initial velocity), it didn't read them in properly, so I had to edit
ctrl_map_ini.F in order to be able to read adxx_files using
openad_active_read_xyz. And after doing so, gradient check works fine. Hope
I did right modifications...
Currently, the set-up is in a local cluster here at Scripps, but talked to
my supervisor and can give you access to it. I guess you will receive it
soon.
Thanks again and looking forward to see whether this problem can be solved.
Regards
Heriberto
On Tue, May 10, 2016 at 7:25 AM, <mitgcm-support-request at mitgcm.org> wrote:
> Send MITgcm-support mailing list submissions to
> mitgcm-support at mitgcm.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mitgcm.org/mailman/listinfo/mitgcm-support
> or, via email, send a message with subject or body 'help' to
> mitgcm-support-request at mitgcm.org
>
> You can reach the person managing the list at
> mitgcm-support-owner at mitgcm.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of MITgcm-support digest..."
>
>
> Today's Topics:
>
> 1. Help OpenAD adjfiles (Heriberto Vazquez)
> 2. Re: Help OpenAD adjfiles (Patrick Heimbach)
> 3. Re: Read NetCDF data (Internal Waves) (Alejandro Jim?nez Rico)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 9 May 2016 10:11:19 -0700
> From: Heriberto Vazquez <heriberto1mx at gmail.com>
> To: mitgcm-support at mitgcm.org
> Subject: [MITgcm-support] Help OpenAD adjfiles
> Message-ID:
> <
> CABO7yNSaXk4EZc92+6pVY0jPT7YbYeHhMYKOCTpuAzUsztr8rA at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hello MITgcm developers
>
> I have a question, particularly to those that have used OpenAD in their
> configurations. Hopefully someone can bring some light to this problem...
>
> Right now I am doing a sensitivity analysis in the Gulf of Mexico using
> OpenAD, working with MITgcm_c65k version. In order to get the sensitivities
> (ADJfiles in TAF) I need to edit and modify the code openad_dumpAdjoint.F
> located on .../pkg/openad/ and of course to modify forward_step.F making a
> Call to openad_dumpAdjoint at the same place where the call to ADJfiles for
> TAF is done (dummy_in_stepping.F or the adjoint version
> addummy_in_stepping.F). Outputs I am getting make perfect sense to me
> because it is something I could expect and some theories about it are
> agreed with it.
>
> Here at Scripps we can use TAF, in order to do a comparison in the outputs
> I did the same implementation but using TAF instead. Everything was great,
> similar patterns and structures in both sensitivities analysis. However the
> differences in values is quite big (i.e. adjetan for TAF values goes from
> -15 to 10 and in OpenAD values goes from -60 to 50) in the other files
> there are big differences as well (i.e. ajduvel, adjvvel, adjsalt, etc.)
> differences are not constant and evolve in time.
>
> Because differences, I was told to use gradient check package an check out
> which one was wrong. After doing some tweaks to generate the proper
> dependencies, I could use gradient check package in both TAF and OpenAD,
> results told me both of them where correct, TAF results were better than
> OpenAD but both correct.
>
> gradient check package works using adxx_files, so I compare adxx_files from
> TAF and OpenAD and results are not exactly the same but are quite similar,
> which led me to think the only problem was in openad_dumpAdjoint.F when
> printing out the adjfiles.
>
> I checked addummy_in_stepping.F and build the same exchanges there but in
> my version of openad_dumpAdjoint.F, of course with the specific subroutines
> for openad (I use openad_exch_rl instead of adexch_rl for example) the
> compilation process finished successfully but at run time I got a
> segmentation fault. This was the error:
>
> [compas-1-5:02282] *** Process received signal ***
> [compas-1-5:02282] Signal: Segmentation fault (11)
> [compas-1-5:02282] Signal code: Address not mapped (1)
> [compas-1-5:02282] Failing at address: 0x563af148
> One time I got it when the backward integration started and other times
> after one or two days of forward integration.
>
> I am not sure if exchanges can be the problem in differences in ADJ files
> because when take out the openad_exchanges, the run is OK and results make
> sense, the values in the evolution from the first backward integration to
> the last one are the ones who doesn't make sense.
>
> Thinking that maybe a normalization in the values can be generating these
> differences, I checked the last value (in backward integration) of adjfiles
> and it is practically the same as in adxx_file (talking about adxx_uvel and
> adjuvel or for vvel) so it is not a normalization or something.
>
> By the way, forward run is almost exactly the same only differences of
> -8xe-13 to 7xe-13 in the values and the cost function is also exactly the
> same, seems to me like only the adjoint integration has different results.
>
> Because of above paragraph, I decided to comment out all parameters at
> data.autodiff file and definitions in inadmode_set and _unset just in case
> one of them (for sure TAF) was changing parameters when backward
> integration is being done. But it didn't work differences are still being
> the same, too big.
>
> So, I cannot yet figure out why those differences exist in the adjfiles.
> Has someone faced the same problem? Some ideas what could be causing those
> differences? How can I solve it? Hope the problem description is clear
> enough, if not please feel free and ask.
>
> Thank you very much
>
> Best regards,
>
> Heriberto
> Postdoc SIO-UCSD
>
> __
> No podemos resolver problemas usando el mismo tipo de pensamiento que
> usamos cuando los creamos...
> Einstein
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://mitgcm.org/pipermail/mitgcm-support/attachments/20160509/33dac41c/attachment-0001.htm
> >
>
> ------------------------------
>
> Message: 2
> Date: Mon, 9 May 2016 14:56:04 -0400
> From: Patrick Heimbach <heimbach at mit.edu>
> To: mitgcm-support at mitgcm.org
> Subject: Re: [MITgcm-support] Help OpenAD adjfiles
> Message-ID: <6C712F17-C571-4940-A4CC-8B76EF25574D at mit.edu>
> Content-Type: text/plain; charset="windows-1252"
>
> Hi Heriberto,
>
> well, it looks like the good news is that you report the last adjoint
> value (going backwards) to be practically the same between both tools and
> gradient checks seemed ok too (not sure why you had to tweak the gradient
> check package, it?s implemented - and used - for both tools). Almost the
> only way the last values can be ok is for intermediary values to be ok too,
> and so most likely some issue with I/O of intermediate values.
>
> Is your setup easily accessible somewhere (on a common machine, e.g. TACC
> or NCAR)?
> Could take a look at some point soon - although currently a bit swamped.
>
> -Patrick
>
> On May 9, 2016, at 1:11 PM, Heriberto Vazquez <heriberto1mx at gmail.com>
> wrote:
>
> > Hello MITgcm developers
> >
> > I have a question, particularly to those that have used OpenAD in their
> configurations. Hopefully someone can bring some light to this problem...
> >
> > Right now I am doing a sensitivity analysis in the Gulf of Mexico using
> OpenAD, working with MITgcm_c65k version. In order to get the sensitivities
> (ADJfiles in TAF) I need to edit and modify the code openad_dumpAdjoint.F
> located on .../pkg/openad/ and of course to modify forward_step.F making a
> Call to openad_dumpAdjoint at the same place where the call to ADJfiles for
> TAF is done (dummy_in_stepping.F or the adjoint version
> addummy_in_stepping.F). Outputs I am getting make perfect sense to me
> because it is something I could expect and some theories about it are
> agreed with it.
> >
> > Here at Scripps we can use TAF, in order to do a comparison in the
> outputs I did the same implementation but using TAF instead. Everything was
> great, similar patterns and structures in both sensitivities analysis.
> However the differences in values is quite big (i.e. adjetan for TAF values
> goes from -15 to 10 and in OpenAD values goes from -60 to 50) in the other
> files there are big differences as well (i.e. ajduvel, adjvvel, adjsalt,
> etc.) differences are not constant and evolve in time.
> >
> > Because differences, I was told to use gradient check package an check
> out which one was wrong. After doing some tweaks to generate the proper
> dependencies, I could use gradient check package in both TAF and OpenAD,
> results told me both of them where correct, TAF results were better than
> OpenAD but both correct.
> >
> > gradient check package works using adxx_files, so I compare adxx_files
> from TAF and OpenAD and results are not exactly the same but are quite
> similar, which led me to think the only problem was in openad_dumpAdjoint.F
> when printing out the adjfiles.
> >
> > I checked addummy_in_stepping.F and build the same exchanges there but
> in my version of openad_dumpAdjoint.F, of course with the specific
> subroutines for openad (I use openad_exch_rl instead of adexch_rl for
> example) the compilation process finished successfully but at run time I
> got a segmentation fault. This was the error:
> >
> > [compas-1-5:02282] *** Process received signal ***
> > [compas-1-5:02282] Signal: Segmentation fault (11)
> > [compas-1-5:02282] Signal code: Address not mapped (1)
> > [compas-1-5:02282] Failing at address: 0x563af148
> > One time I got it when the backward integration started and other times
> after one or two days of forward integration.
> >
> > I am not sure if exchanges can be the problem in differences in ADJ
> files because when take out the openad_exchanges, the run is OK and results
> make sense, the values in the evolution from the first backward integration
> to the last one are the ones who doesn't make sense.
> >
> > Thinking that maybe a normalization in the values can be generating
> these differences, I checked the last value (in backward integration) of
> adjfiles and it is practically the same as in adxx_file (talking about
> adxx_uvel and adjuvel or for vvel) so it is not a normalization or
> something.
> >
> > By the way, forward run is almost exactly the same only differences of
> -8xe-13 to 7xe-13 in the values and the cost function is also exactly the
> same, seems to me like only the adjoint integration has different results.
> >
> > Because of above paragraph, I decided to comment out all parameters at
> data.autodiff file and definitions in inadmode_set and _unset just in case
> one of them (for sure TAF) was changing parameters when backward
> integration is being done. But it didn't work differences are still being
> the same, too big.
> >
> > So, I cannot yet figure out why those differences exist in the adjfiles.
> Has someone faced the same problem? Some ideas what could be causing those
> differences? How can I solve it? Hope the problem description is clear
> enough, if not please feel free and ask.
> >
> > Thank you very much
> >
> > Best regards,
> >
> > Heriberto
> > Postdoc SIO-UCSD
> >
> > __
> > No podemos resolver problemas usando el mismo tipo de pensamiento que
> usamos cuando los creamos...
> > Einstein
> > _______________________________________________
> > MITgcm-support mailing list
> > MITgcm-support at mitgcm.org
> > http://mitgcm.org/mailman/listinfo/mitgcm-support
>
>
> --------------------------------------------------------
> Patrick Heimbach, Ph.D. | http://heimbach.wordpress.com
>
> * The University of Texas at Austin *
> The Institute for Computational Engineering and Sciences
> Institute for Geophysics | Jackson School of Geosciences
> 201 East 24th Street, POB 4.232 | Austin, TX 78712 | USA
> FON: +1-512-232-7694 | Email: heimbach at utexas.edu
>
> * Massachusetts Institute of Technology *
> Department of Earth, Atmospheric, and Planetary Sciences
> 77 Massachusetts Ave, 54-1420 | Cambridge MA 02139 | USA
> FON: +1-617-253-5259 | Email: heimbach at mit.edu
> --------------------------------------------------------
>
>
>
>
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: smime.p7s
> Type: application/pkcs7-signature
> Size: 1845 bytes
> Desc: not available
> URL: <
> http://mitgcm.org/pipermail/mitgcm-support/attachments/20160509/ea7e9c7d/attachment-0001.p7s
> >
>
> ------------------------------
>
> Message: 3
> Date: Tue, 10 May 2016 14:25:00 +0000
> From: Alejandro Jim?nez Rico <paulexca at hotmail.com>
> To: "mitgcm-support at mitgcm.org" <mitgcm-support at mitgcm.org>
> Subject: Re: [MITgcm-support] Read NetCDF data (Internal Waves)
> Message-ID: <DUB127-W87063E89799A67E84F0BFDD8710 at phx.gbl>
> Content-Type: text/plain; charset="windows-1252"
>
> I am using python.
> Look, I wrote this:
> from matplotlib import pyplot as plt
> import pandas as pd
> import numpy as np
> import netCDF4
> fp='toread.nc'
> nc = netCDF4.Dataset(fp)
> print "Stationary or animated? (Say 'st' or 'a')"
> q1 = raw_input()
> print "Filled contours or an image? (Say 'fc' or 'image')"
> q2 = raw_input()
> print(nc['Temp'].shape)
> t = 1
> y = 0
> if q1 == 'a':
> while t< 10000:
> if q2 == 'image':
> plt.imshow(nc['Temp'][t,:,y,:])
> if q2 == 'fc':
> a=nc['Temp'][t,:,y,:]
> b=np.flipup(a)
> plt.contourf(nc['Temp'][t,:,y,:])
> plt.pause(.001)
> t=t+1
> plt.draw()
> if q1 == 'st':
> if q2 == 'image':
> plt.imshow(nc['Temp'][t,:,y,:])
> if q2 == 'fc':
> a=nc['Temp'][t,:,y,:]
> b=np.flipup(a)
> plt.contourf(nc['Temp'][t,:,y,:])
> plt.show()But now I would like to not just looking the data, I want to
> touch it a bit.
> I assume that you know that the position coordinates are no centralized;
> well, I would like to centralize it so then I will can calculate more
> interesting things like (for example) the kinetic energy of the fluid
> particles and plot it.
> How could I do it? Do that things is what - essentially - I am trying to
> learn about.
> Again, really thank you for your help;Alejandro
> From: jklymak at uvic.ca
> Date: Mon, 2 May 2016 08:53:40 -0700
> To: mitgcm-support at mitgcm.org
> Subject: Re: [MITgcm-support] Read NetCDF data (Internal Waves)
>
> What package are you using to do the analysis? Matlab and python are the
> most common, and they both have excellent netcdf libraries.
> Cheers, Jody
>
>
> On May 2, 2016, at 2:59 AM, Alejandro Jim?nez Rico <paulexca at hotmail.com>
> wrote:
>
> From: paulexca at hotmail.com
> To: mitgcm-support at mitgcm.org
> Subject: RE: [MITgcm-support] Read NetCDF data (Internal Waves)
> Date: Mon, 2 May 2016 09:59:24 +0000
>
> Hello Roland,
> Thank you for answering me. Actually I am trying to completely understand
> the Internal Waves model to carry out my own simulations using it.
> First of all I would like to get the full matrix data of the .nc output of
> this model; so I'll can read it in a different ways and more personalized
> than using ncview (for example, I would like to see the time evolution of
> velocity in a precise point of the space, like field experiments).
> Moreover, I'm trying to get the scale of this model (miles, kilometers,
> seconds, etc.) and all the information I can find about the performance of
> the model.
> I hope you can help me and - again - really thank you for your help.
> Alejandro
>
> From: Roland.Young at physics.ox.ac.uk
> To: mitgcm-support at mitgcm.org
> Date: Wed, 20 Apr 2016 11:30:55 +0000
> Subject: Re: [MITgcm-support] Read NetCDF data (Internal Waves)
>
> Hi Alejandro,
> I?m not entirely sure what you are trying to do; ncdump is useful for
> looking inside a NetCDF file but not for manipulating its contents, which
> is what I think you are trying to do.
> If you want to view the fields themselves I suggest using ncview, which is
> very useful for doing this quickly. There are other programs that can read
> in NetCDF files for viewing, such as xconv and Panoply (the second of which
> is very powerful).
> ncdump -c $file should print the values of all the grid variables in the
> .nc file, typically X, Y, Z, T for longitude, latitude, pressure/height,
> time.
> ncdump -h $file will just print the headers (i.e. a list of variables in
> the file).
> ncdump -v <variable> $file will print the complete record for a single
> variable to the screen. For example, ncdump -v U $file will print the
> entire zonal velocity field in the .nc file to the screen. This is not
> particularly useful if you want to manipulate the data, however.
> You can manipulate NetCDF files directly using the NCO library:
> http://nco.sourceforge.net
> However, most useful for manipulating the data itself rather than just
> view it using ncview is an interpreter such as IDL, Matlab, or Python. Each
> of these have libraries to read NetCDF files, after which you can do what
> you like with the data.
> Don?t forget that if you are running a parallel model run you will need to
> stitch the tiles together using gluemncbig, otherwise each .nc file only
> contains one tile?s worth of data.
> Hope that helps,
> Roland
>
>
>
> On 20 Apr 2016, at 11:52, Alejandro Jim?nez Rico <paulexca at hotmail.com>
> wrote:Hello, I am trying to figure out how to read all the netcdf data
> using ncdump. I did $ ndump -c filetoread.nc
> But I would want to organize all this data a little bit. How do I do this?
> I don't find information about the dimensions or the variables of this
> NetCDF in the mitgcm documentation.
> Thank you,Alejandro_______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
> _______________________________________________ MITgcm-support mailing
> list MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support_______________________________________________MITgcm-support
> mailing listMITgcm-support at mitgcm.orghttp://
> mitgcm.org/mailman/listinfo/mitgcm-support
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://mitgcm.org/pipermail/mitgcm-support/attachments/20160510/6ca3deb8/attachment.htm
> >
>
> ------------------------------
>
> _______________________________________________
> MITgcm-support mailing list
> MITgcm-support at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-support
>
>
> End of MITgcm-support Digest, Vol 155, Issue 5
> **********************************************
>
--
Saludos
Heriberto
__
No podemos resolver problemas usando el mismo tipo de pensamiento que
usamos cuando los creamos...
Einstein
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-support/attachments/20160510/88075ef1/attachment-0001.htm>
More information about the MITgcm-support
mailing list