[MITgcm-devel] migration to github and readthedocs

Fri Jul 21 06:56:53 EDT 2017

Thanks Martin.

Headings:
I chose 4 depth levels for numbered headings because that was what I saw in most of the manual. We can set more, but this has to be set globally in the `index.rst` file. You make a strong case for increasing the depth of numbered sections - I’ll put in a pull request for that change when I get a chance.

Workflow:
I agree, that branching and putting in a pull request adds two steps to the process. However, as we all become more familiar with git and the workflow, I think that these two steps will begin to fade into the background. Creating a new branch will simply be part of the development workflow, regardless of whether one is working on the code or the manual.

I also agree that having the manual in the repo is a good idea, for a couple of reasons: I’m not sure that there is currently any clear process for general users to contribute to the manual (I don’t expect that they will document entire packages, but they will hopefully add clarifications, fix typos, and possibly even update sections); and combined with the PR work flow we can politely ask people to update documentation if their proposed code changes require it.

Citations:
This is definitely a pain. I’m still working on figuring out an automated way of converting that brings them across. If/when that happens, the details will go in the manual.

Manual editing and previewing:
I’m afraid I don’t know of any WYSIWYG editors for sphinx/rst. I’ve been working in a text editor and periodically building the output and refreshing a tab with the page. I guess because I’m more familiar with it, I don’t have to build as often, and haven’t found this too tedious. Sorry I don’t have any better suggestions.

.bib file:
Thanks for putting those references into it.

Finally:
Thank you for your efforts - learning about which parts of the process you are finding tedious or difficult is very helpful. I’ll try and find ways to ease the pain points you’ve identified.

Cheers,
Ed

> On 20 Jul 2017, at 11:01, Martin Losch <Martin.Losch at awi.de> wrote:
> 
> Hi Ed,
> 
> I have now added my incomplete “second pass” as a pull request, actually more than one, I am afraid.
> 
> I have a few comments below, but more generally, I see an issue with the section numbering. In the tex-manual we have very many levels of subsections with section numbers. Most of these are turned into unnumbers and hence unlabels “paragraphs”, e.g. in seaice.rst only 3 numbered sections remain (and the paragraphs do not show up in the list in the left hand side of the page, but again, maybe my mistake). That makes it very hard to refer users to specific parts of the manual (“go to section 2.6.2.3 Description (seaice) to read about …” is not very accurate as 2.6.2.3 has currently 12 subsection (subsub[…]subsections). obcs is similar. Are there any ideas about that? I don’t …
> 
> My feeling is that for the manual, the workflow of forking, branching, making a PR is really very tedious if you want to work and change a lot on the text, but I guess there is no better way if the manual is included in the repo (which I think is a great idea!!). 
> 
> Martin
> 
>> On 19. Jul 2017, at 19:02, Edward Doddridge <edoddridge at gmail.com> wrote:
>> 
>> Hi Martin,
>> 
>> I’m sorry to hear you’ve been having difficulties. As the others have already said, we want to hear about the problems and issues that you encounter, so complaining is absolutely helpful.
>> 
>>> - In the conversion lots of latex code does not get interpreted, probably my fault, because I used makros like “\code{}” which seem to be stripped from the text in many places (along with it’s argument!!). Also many symbols that I use routinely in equations in this section (usually something that I defined with “\newcommand”) are obviously replaced with unpredictable outcome, so that I will have to check many equations. This will require comparing the tex and rst sources very carefully and will take a lot of time and nerves, especially because debugging a markup language that I haven’t used before is terribly tedious.
>> 
>> This is an issue with the conversion - ReStructured Text is a much less complex markup language and doesn’t support macros. I haven’t  yet found a way to automate their conversion, so fixing  the issues you've identified has been part of the second pass (a manual  clean up after the initial automatic conversion).
>> 
>> Hopefully the macro issue won’t be too much of a problem going forward, since most journals don’t like them either.
> The macro stuff may probably not be a problem down the road, but I assume that the referencing scheme will be a pain. Many journals (JGR, JPO, Ocean Modelling, …) do support and even require the natbib -\citet and \citep (and more) commands which get totally ignored by the conversion.
>> 
>>> - I cannot make the equation referencing work reliably (sometimes it works, sometimes it doesn’t), please point me to a working example or a rst reference that you can recommend.
>> 
>> The equation reference on the barotropic gyre example page seems to work. Perhaps that will help?
> I used those a template to begin with. Sometimes it works, sometimes it doesn’t, probably some spaces or blank lines that are interpreted as commands and that I am not aware of. I am giving up for now. It would be nice for me to have a simpler debugging work flow than “edit in one program && make html in command line && open in browser or reload page in browser && scroll back to where the problem was over and over again”.  Do you have something that you can recommend?
>> 
>> Source:
>> https://raw.githubusercontent.com/altMITgcm/MITgcm66h/master/docs/examples/barotropic_gyre/barotropic_gyre.rst
>> 
>> html output
>> http://altmitgcm66h.readthedocs.io/en/latest/examples/examples.html#barotropic-gyre-mitgcm-example
>> 
>> 
>>> - there are many references missing from the new manual_bibliography.bib. Is this on purpose? Do I have to put things back in mysef or will there be some automatism?
>> 
>> It is on purpose. The documentation wouldn’t build using the .bib file from the current manual. I eventually gave up on trying to work out why and started a new one, copying over the references as needed. As Chris said, we’re planning on doing most of the conversion at MIT, so you don’t have to worry about that.
> I put back the references that I think are required for seaice.rst and re-formatted some of them. For me, the documentation builds fine with the modified file.
>> 
>> If you’ve cleaned the seaice page up a bit, then it would be great if you could put that in as a pull request - there’s no point letting your efforts go to waste.
>> 
>> 
>> Jeff - yes, definitely. We can add a section about using pandoc to convert .tex files to .rst and point out the common pitfalls.
>> 
>> 
>> Ed
>> 
>> 
>> 
>> 
>>> On 19 Jul 2017, at 12:00, Jeffery R Scott <jscott at mit.edu> wrote:
>>> 
>>> Hi Martin,
>>> 
>>> No apologies necessary, complaining is important :)
>>> 
>>> Ed is using pandoc as first-pass at converting the existing doc to .rst; I’m trying to get up to speed to help with that. There is some manual checking/fixing required for most stuff as a second pass.
>>> 
>>> For NEW documentation, I looked at how difficult it was to format a new contribution in .rst formatting, and concluded it was pretty straightforward.
>>> I did not however consider converting existing latex directly into .rst — but Ed, can’t we simply add a guide how to use pandoc to facilitate this process  for people?
>>> 
>>> Jeff
>>> 
>>> 
>>> 
>>>> On Jul 19, 2017, at 11:50 AM, Martin Losch <Martin.Losch at awi.de> wrote:
>>>> 
>>>> Hi Chris,
>>>> sorry for complaining. I just spent all afternoon on trying to make seaice.rst look similar to the corresponding latex result and it is really painful.
>>>> 
>>>> I can commit (pull request) what I have so far if you want, but it’s not complete and I am not planning on continuing much further.
>>>> 
>>>> Martin
>>>> 
>>>>> On 19. Jul 2017, at 17:45, Chris Hill <cnh at mit.edu> wrote:
>>>>> 
>>>>> Hi Martin,
>>>>> 
>>>>> The comments are really helpful.
>>>>> 
>>>>> We do envisioning converting a lot of the existing manual at the
>>>>> MIT end (i.e.not at expense to others in $ or time). At this stage we are
>>>>> looking for feedback and to see what is practical, what is broken, who
>>>>> likes what, what is right way forward, where are the bugs etc....
>>>>> 
>>>>> Chris
>>>>> 
>>>>> On Wed, Jul 19, 2017 at 10:57 AM, Martin Losch <Martin.Losch at awi.de> wrote:
>>>>>> A bit of complaining:
>>>>>> The conversion of the documentation from latex to rst is not as straightforward as I had expected.
>>>>>> 
>>>>>> I am just looking at the seaice.rst file, since that’s the last section that I have actively modified (but also applies to other sections like OBCS):
>>>>>> - In the conversion lots of latex code does not get interpreted, probably my fault, because I used makros like “\code{}” which seem to be stripped from the text in many places (along with it’s argument!!). Also many symbols that I use routinely in equations in this section (usually something that I defined with “\newcommand”) are obviously replaced with unpredictable outcome, so that I will have to check many equations. This will require comparing the tex and rst sources very carefully and will take a lot of time and nerves, especially because debugging a markup language that I haven’t used before is terribly tedious.
>>>>>> - I cannot make the equation referencing work reliably (sometimes it works, sometimes it doesn’t), please point me to a working example or a rst reference that you can recommend.
>>>>>> - there are many references missing from the new manual_bibliography.bib. Is this on purpose? Do I have to put things back in mysef or will there be some automatism?
>>>>>> 
>>>>>> Together with this new work flow, which seem to make the turnaround much slower (at least for the unflexible me), this makes writing the documentation much more time consuming and less easy to do, because I will not be able to just cut and paste from other latex source (papers). On top, I don’t feel like repeating a lot of the work that I have already done. I am a little disappointed by this and just hope that the transition will be easier for the actual code development.
>>>>>> 
>>>>>> Martin
>>>>>> 
>>>>>>> On 18. Jul 2017, at 19:54, Jean-Michel Campin <jmc at mit.edu> wrote:
>>>>>>> 
>>>>>>> Hi All,
>>>>>>> 
>>>>>>> We have been doing some preparatory work to migrate the MITgcm code
>>>>>>> to a git repository at github. A test/preview setup for this is at
>>>>>>> https://github.com/altmitgcm/MITgcm66h. If you have a chance to take a
>>>>>>> look it would be great to get any feedback at this stage. Some things
>>>>>>> to note
>>>>>>> 
>>>>>>> o The github setup is a preview.
>>>>>>> 
>>>>>>> o Pending feedback, we are currently on track to switch to using
>>>>>>> https://github.com/mitgcm/mitgcm_core as the  master repository for
>>>>>>> distribution and development.
>>>>>>> 
>>>>>>> o The switch is expected to happen in September, if everything go well.
>>>>>>> 
>>>>>>> o When this switch happens the CVS repository for core code and
>>>>>>> documentation will be frozen.
>>>>>>> 
>>>>>>> o History and change information will be carried over to github.
>>>>>>> 
>>>>>>> o The MITgcm_contrib CVS repository will remain active now, although
>>>>>>> several things in MITgcm_contrib have already "self-migrated" to git.
>>>>>>> 
>>>>>>> There are some changes that will go with the switch to git around how
>>>>>>> to work with MITgcm code. For development there is a primer here
>>>>>>> http://altmitgcm66h.readthedocs.io/en/latest/contributing.html#quickstart-guide
>>>>>>> that describes how development workflows will usually function. The
>>>>>>> full development workflow entails user accounts set up with github.com
>>>>>>> (https://github.com), travis.org (https://travis-ci.org) (which
>>>>>>> automates some testreport checking - see
>>>>>>> http://altmitgcm66h.readthedocs.io/en/latest/contributing.html#automatic-testing-with-travis-ci
>>>>>>> and https://github.com/altMITgcm/MITgcm66h/blob/master/.travis.yml)
>>>>>>> and readthedocs.io (https://readthedocs.org). The short form of the
>>>>>>> workflow, in github speak, is
>>>>>>> o fork the official repo
>>>>>>> o clone the fork
>>>>>>> o update code
>>>>>>> o push to clone and test through travis
>>>>>>> o submit pull request
>>>>>>> o thing gets merged, assuming they pass inspection by testreport and a
>>>>>>> master repository committer (a human!). Chris, Oliver and Jean-Michel
>>>>>>> are the committers at present. This will evolve as we learn what we
>>>>>>> are doing!
>>>>>>> 
>>>>>>> The other noticeable change is around the MITgcm manual which we are
>>>>>>> planning to migrate to a format compatible with readthedocs.org. This
>>>>>>> format can produce online and hard copy documentation. It also
>>>>>>> integrates with development tools in github, making it easier to see
>>>>>>> updates from edits in "realtime". The new documentation format
>>>>>>> (restructured text or .rst) should be recognizable to anyone who
>>>>>>> speaks latex and much latex content carries over. However, syntax for
>>>>>>> section headings, labels, citations etc... is slightly different.
>>>>>>> Actual equation writing syntax is generally almost the same as latex.
>>>>>>> Ed Doddridge plus Jeff Scott have been doing a fantastic job providing
>>>>>>> examples of changed manual sections. These show how the syntax differs
>>>>>>> and what end product looks like.
>>>>>>> 
>>>>>>> For this mail that is probably enough. As stated earlier it would be
>>>>>>> great to get feedback! We are planning to hold a regular WebEx call
>>>>>>> starting in September, as another way to gather input on this and
>>>>>>> other planned activities.
>>>>>>> 
>>>>>>> Cheers,
>>>>>>> 
>>>>>>> Chris and Jean-Michel
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> MITgcm-devel mailing list
>>>>>>> MITgcm-devel at mitgcm.org
>>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> MITgcm-devel mailing list
>>>>>> MITgcm-devel at mitgcm.org
>>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>>> 
>>>>> _______________________________________________
>>>>> MITgcm-devel mailing list
>>>>> MITgcm-devel at mitgcm.org
>>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>>> 
>>>> 
>>>> _______________________________________________
>>>> MITgcm-devel mailing list
>>>> MITgcm-devel at mitgcm.org
>>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>>> 
>>> _______________________________________________
>>> MITgcm-devel mailing list
>>> MITgcm-devel at mitgcm.org
>>> http://mitgcm.org/mailman/listinfo/mitgcm-devel
>> 
>> _______________________________________________
>> MITgcm-devel mailing list
>> MITgcm-devel at mitgcm.org <mailto:MITgcm-devel at mitgcm.org>
>> http://mitgcm.org/mailman/listinfo/mitgcm-devel <http://mitgcm.org/mailman/listinfo/mitgcm-devel>
> 
> 
> _______________________________________________
> MITgcm-devel mailing list
> MITgcm-devel at mitgcm.org <mailto:MITgcm-devel at mitgcm.org>
> http://mitgcm.org/mailman/listinfo/mitgcm-devel <http://mitgcm.org/mailman/listinfo/mitgcm-devel>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mitgcm.org/pipermail/mitgcm-devel/attachments/20170721/6e8b7536/attachment-0001.htm>