[MITgcm-devel] migration to github and readthedocs

Fri Jul 21 15:00:40 EDT 2017

>
> My feeling is that for the manual, the workflow of forking, branching,
> making a PR is really very tedious if you want to work and change a lot on
> the text, but I guess there is no better way if the manual is included in
> the repo (which I think is a great idea!!).
>

I just wanted to chime in with a counterpoint. For me, the process of
contributing to the docs has always seemed intimidating. I couldn't figure
out how to build them locally, and I was reluctant to commit changes
"blind" without previewing them first. ReST, Sphinx, and readthedocs is a
widely used and familiar format and platform for many people, including
myself. So in the future, I think I will be more likely to contribute to
the docs if we can manage to migrate them.

But at the same time, I recognize that the friction and frustration Martin
experienced is a major downside. Hopefully a balance can be found between
evolving to a more modern platform and not alienating the most productive
and enthusiastic MITgcm contributors! Ed had some great suggestions on how
to achieve that.

> Martin
>
> > On 19. Jul 2017, at 19:02, Edward Doddridge <edoddridge at gmail.com>
> wrote:
> >
> > Hi Martin,
> >
> > I’m sorry to hear you’ve been having difficulties. As the others have
> already said, we want to hear about the problems and issues that you
> encounter, so complaining is absolutely helpful.
> >
> >> - In the conversion lots of latex code does not get interpreted,
> probably my fault, because I used makros like “\code{}” which seem to be
> stripped from the text in many places (along with it’s argument!!). Also
> many symbols that I use routinely in equations in this section (usually
> something that I defined with “\newcommand”) are obviously replaced with
> unpredictable outcome, so that I will have to check many equations. This
> will require comparing the tex and rst sources very carefully and will take
> a lot of time and nerves, especially because debugging a markup language
> that I haven’t used before is terribly tedious.
> >
> > This is an issue with the conversion - ReStructured Text is a much less
> complex markup language and doesn’t support macros. I haven’t  yet found a
> way to automate their conversion, so fixing  the issues you've identified
> has been part of the second pass (a manual  clean up after the initial
> automatic conversion).
> >
> > Hopefully the macro issue won’t be too much of a problem going forward,
> since most journals don’t like them either.
> The macro stuff may probably not be a problem down the road, but I assume
> that the referencing scheme will be a pain. Many journals (JGR, JPO, Ocean
> Modelling, …) do support and even require the natbib -\citet and \citep
> (and more) commands which get totally ignored by the conversion.
> >
> >> - I cannot make the equation referencing work reliably (sometimes it
> works, sometimes it doesn’t), please point me to a working example or a rst
> reference that you can recommend.
> >
> > The equation reference on the barotropic gyre example page seems to
> work. Perhaps that will help?
> I used those a template to begin with. Sometimes it works, sometimes it
> doesn’t, probably some spaces or blank lines that are interpreted as
> commands and that I am not aware of. I am giving up for now. It would be
> nice for me to have a simpler debugging work flow than “edit in one program
> && make html in command line && open in browser or reload page in browser
> && scroll back to where the problem was over and over again”.  Do you have
> something that you can recommend?
> >
> > Source:
> > https://raw.githubusercontent.com/altMITgcm/MITgcm66h/
> master/docs/examples/barotropic_gyre/barotropic_gyre.rst
> >
> > html output
> > http://altmitgcm66h.readthedocs.io/en/latest/examples/examples.html#
> barotropic-gyre-mitgcm-example
> >
> >
> >> - there are many references missing from the new
> manual_bibliography.bib. Is this on purpose? Do I have to put things back
> in mysef or will there be some automatism?
> >
> > It is on purpose. The documentation wouldn’t build using the .bib file
> from the current manual. I eventually gave up on trying to work out why and
> started a new one, copying over the references as needed. As Chris said,
> we’re planning on doing most of the conversion at MIT, so you don’t have to
> worry about that.
> I put back the references that I think are required for seaice.rst and
> re-formatted some of them. For me, the documentation builds fine with the
> modified file.
> >
> > If you’ve cleaned the seaice page up a bit, then it would be great if
> you could put that in as a pull request - there’s no point letting your
> efforts go to waste.
> >
> >
> > Jeff - yes, definitely. We can add a section about using pandoc to
> convert .tex files to .rst and point out the common pitfalls.
> >
> >
> > Ed
> >
> >
> >
> >
> >> On 19 Jul 2017, at 12:00, Jeffery R Scott <jscott at mit.edu> wrote:
> >>
> >> Hi Martin,
> >>
> >> No apologies necessary, complaining is important :)
> >>
> >> Ed is using pandoc as first-pass at converting the existing doc to
> .rst; I’m trying to get up to speed to help with that. There is some manual
> checking/fixing required for most stuff as a second pass.
> >>
> >> For NEW documentation, I looked at how difficult it was to format a new
> contribution in .rst formatting, and concluded it was pretty
> straightforward.
> >> I did not however consider converting existing latex directly into .rst
> — but Ed, can’t we simply add a guide how to use pandoc to facilitate this
> process  for people?
> >>
> >> Jeff
> >>
> >>
> >>
> >>> On Jul 19, 2017, at 11:50 AM, Martin Losch <Martin.Losch at awi.de>
> wrote:
> >>>
> >>> Hi Chris,
> >>> sorry for complaining. I just spent all afternoon on trying to make
> seaice.rst look similar to the corresponding latex result and it is really
> painful.
> >>>
> >>> I can commit (pull request) what I have so far if you want, but it’s
> not complete and I am not planning on continuing much further.
> >>>
> >>> Martin
> >>>
> >>>> On 19. Jul 2017, at 17:45, Chris Hill <cnh at mit.edu> wrote:
> >>>>
> >>>> Hi Martin,
> >>>>
> >>>> The comments are really helpful.
> >>>>
> >>>> We do envisioning converting a lot of the existing manual at the
> >>>> MIT end (i.e.not at expense to others in $ or time). At this stage we
> are
> >>>> looking for feedback and to see what is practical, what is broken, who
> >>>> likes what, what is right way forward, where are the bugs etc....
> >>>>
> >>>> Chris
> >>>>
> >>>> On Wed, Jul 19, 2017 at 10:57 AM, Martin Losch <Martin.Losch at awi.de>
> wrote:
> >>>>> A bit of complaining:
> >>>>> The conversion of the documentation from latex to rst is not as
> straightforward as I had expected.
> >>>>>
> >>>>> I am just looking at the seaice.rst file, since that’s the last
> section that I have actively modified (but also applies to other sections
> like OBCS):
> >>>>> - In the conversion lots of latex code does not get interpreted,
> probably my fault, because I used makros like “\code{}” which seem to be
> stripped from the text in many places (along with it’s argument!!). Also
> many symbols that I use routinely in equations in this section (usually
> something that I defined with “\newcommand”) are obviously replaced with
> unpredictable outcome, so that I will have to check many equations. This
> will require comparing the tex and rst sources very carefully and will take
> a lot of time and nerves, especially because debugging a markup language
> that I haven’t used before is terribly tedious.
> >>>>> - I cannot make the equation referencing work reliably (sometimes it
> works, sometimes it doesn’t), please point me to a working example or a rst
> reference that you can recommend.
> >>>>> - there are many references missing from the new
> manual_bibliography.bib. Is this on purpose? Do I have to put things back
> in mysef or will there be some automatism?
> >>>>>
> >>>>> Together with this new work flow, which seem to make the turnaround
> much slower (at least for the unflexible me), this makes writing the
> documentation much more time consuming and less easy to do, because I will
> not be able to just cut and paste from other latex source (papers). On top,
> I don’t feel like repeating a lot of the work that I have already done. I
> am a little disappointed by this and just hope that the transition will be
> easier for the actual code development.
> >>>>>
> >>>>> Martin
> >>>>>
> >>>>>> On 18. Jul 2017, at 19:54, Jean-Michel Campin <jmc at mit.edu> wrote:
> >>>>>>
> >>>>>> Hi All,
> >>>>>>
> >>>>>> We have been doing some preparatory work to migrate the MITgcm code
> >>>>>> to a git repository at github. A test/preview setup for this is at
> >>>>>> https://github.com/altmitgcm/MITgcm66h. If you have a chance to
> take a
> >>>>>> look it would be great to get any feedback at this stage. Some
> things
> >>>>>> to note
> >>>>>>
> >>>>>> o The github setup is a preview.
> >>>>>>
> >>>>>> o Pending feedback, we are currently on track to switch to using
> >>>>>> https://github.com/mitgcm/mitgcm_core as the  master repository for
> >>>>>> distribution and development.
> >>>>>>
> >>>>>> o The switch is expected to happen in September, if everything go
> well.
> >>>>>>
> >>>>>> o When this switch happens the CVS repository for core code and
> >>>>>> documentation will be frozen.
> >>>>>>
> >>>>>> o History and change information will be carried over to github.
> >>>>>>
> >>>>>> o The MITgcm_contrib CVS repository will remain active now, although
> >>>>>> several things in MITgcm_contrib have already "self-migrated" to
> git.
> >>>>>>
> >>>>>> There are some changes that will go with the switch to git around
> how
> >>>>>> to work with MITgcm code. For development there is a primer here
> >>>>>> http://altmitgcm66h.readthedocs.io/en/latest/
> contributing.html#quickstart-guide
> >>>>>> that describes how development workflows will usually function. The
> >>>>>> full development workflow entails user accounts set up with
> github.com
> >>>>>> (https://github.com), travis.org (https://travis-ci.org) (which
> >>>>>> automates some testreport checking - see
> >>>>>> http://altmitgcm66h.readthedocs.io/en/latest/
> contributing.html#automatic-testing-with-travis-ci
> >>>>>> and https://github.com/altMITgcm/MITgcm66h/blob/master/.travis.yml)
> >>>>>> and readthedocs.io (https://readthedocs.org). The short form of the
> >>>>>> workflow, in github speak, is
> >>>>>> o fork the official repo
> >>>>>> o clone the fork
> >>>>>> o update code
> >>>>>> o push to clone and test through travis
> >>>>>> o submit pull request
> >>>>>> o thing gets merged, assuming they pass inspection by testreport
> and a
> >>>>>> master repository committer (a human!). Chris, Oliver and
> Jean-Michel
> >>>>>> are the committers at present. This will evolve as we learn what we
> >>>>>> are doing!
> >>>>>>
> >>>>>> The other noticeable change is around the MITgcm manual which we are
> >>>>>> planning to migrate to a format compatible with readthedocs.org.
> This
> >>>>>> format can produce online and hard copy documentation. It also
> >>>>>> integrates with development tools in github, making it easier to see
> >>>>>> updates from edits in "realtime". The new documentation format
> >>>>>> (restructured text or .rst) should be recognizable to anyone who
> >>>>>> speaks latex and much latex content carries over. However, syntax
> for
> >>>>>> section headings, labels, citations etc... is slightly different.
> >>>>>> Actual equation writing syntax is generally almost the same as
> latex.
> >>>>>> Ed Doddridge plus Jeff Scott have been doing a fantastic job
> providing
> >>>>>> examples of changed manual sections. These show how the syntax
> differs
> >>>>>> and what end product looks like.
> >>>>>>
> >>>>>> For this mail that is probably enough. As stated earlier it would be
> >>>>>> great to get feedback! We are planning to hold a regular WebEx call
> >>>>>> starting in September, as another way to gather input on this and
> >>>>>> other planned activities.
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> Chris and Jean-Michel
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> MITgcm-devel mailing list
> >>>>>> MITgcm-devel at mitgcm.org
> >>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >>>>>
> >>>>>
> >>>>> _______________________________________________
> >>>>> MITgcm-devel mailing list
> >>>>> MITgcm-devel at mitgcm.org
> >>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >>>>
> >>>> _______________________________________________
> >>>> MITgcm-devel mailing list
> >>>> MITgcm-devel at mitgcm.org
> >>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >>>
> >>>
> >>> _______________________________________________
> >>> MITgcm-devel mailing list
> >>> MITgcm-devel at mitgcm.org
> >>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >>
> >> _______________________________________________
> >> MITgcm-devel mailing list
> >> MITgcm-devel at mitgcm.org
> >> http://mitgcm.org/mailman/listinfo/mitgcm-devel
> >
> > _______________________________________________
> > MITgcm-devel mailing list
> > MITgcm-devel at mitgcm.org
> > http://mitgcm.org/mailman/listinfo/mitgcm-devel
>
>
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org
> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-devel/attachments/20170721/2cdf2f7a/attachment-0001.htm>