CppGen - Ugur Buyukdurak

Project Background

Project Definition: A project that aims to shorten C++ development time by automatic code generation. This project targets people who don't have access to full-featured C++ IDEs on the market. C++ language has some strict syntax rules with which not everybody can do everything right every time. This, of course, would mean compiler errors and frustrated programmers. Even if a programmer typed everything correctly, he would still be wasting his time with extra typing because of C++ header and implementation file mechanism. More typing means more work, and the more it gets longer, the more mistakes can be made along the way. Automatic code generation zeros the chance of making mistakes when writing implementation(.cpp) of a declaration file(.h) unless base header file is wrong.

Starting Point: I originate this idea from CS 240 Data Structures Class where I had to code in a complete Linux environment without a full featured IDE. There was no syntax highlighting, type checking or anything similar. There was just a raw text editor and a command line compiler. Compiler would always complain I typed something wrong, and say I needed to fix something. Problems were mainly caused by some variables or methods of a class instance that didn't match what it was declared as in another file. So I thought, generating some source code from base code would ease my job a lot further so I could focus on what I was doing.

The Real Work: Although I had the idea that generating cpp (C++ implementation files) from header files would be great, implementing the project was a serious challenge for me. Even though I had the idea in mind, there were lots of gaps in my mind that had to be filled. Was I going to implement my own compiler, or was I going to use existing tools to achieve what I wanted? If I wanted to implement my own compiler, how could I do that or if I wanted to use existing tools what were they?

The real work has never been coding since coding is all about knowing particular syntax rules and general flow of a programming language with some fundamental understanding. Real challenge was to figure out ways in which intended goal could be accomplished which required long hours wasted for finding right tools.

Another big challenge after figuring out how something could be done was to understand how these "right tools" worked. With several months spent trying to understand a Python library and a program both related to dumping XML of a C++ file and reading it the XML dump file, I was finally ready to do something useful and real. It wasn't "in mind" anymore. It had already gone out of mind.

After 3-4 months of research and understanding, I was finally able to code the project. The time it took for me to code the project was just "one" month. It was actually much shorter but I had other things I did in the meantime.

To develop something is a challenge but to deploy it is another challenge. By knowing just pure code doesn't help to use an operating system efficiently and correctly. Computers are all about operating systems. So it is crucial to know to use them in a way that fits your needs. I had completed courses and received two certification for Linux System Administration before I ever needed to deploy my project on the web. The skills I have gained during the System Administration really helped me manage my project efficiently. I was able to make it online in just one day.

The point I am trying to make is that it wasn't hard coding or deploying the project, but hard part was the time passed researching and learning and gathering all these different things to bring one meaningful thing into existence.

What I learned: I had the chance to develop my System Administration skills a lot since I had to deal with ssh, Linux File System, Web Servers and the Linux operating system itself. Besides those, I also had to have fundamental understanding of DNS system, TCP/IP interface, Python Web Frameworks, Python templating Engines to create automated text, several Python libraries, so forth and so on. But most importantly I have realized that hardest part is never coding but rather analyzing.

One important thing that I didn't mention before is importance of a design flow of a program. It is important to design a program in a way in which it can be extended and modified easily. Time, requirements and demands all change and so should the program without a need to recreate it from the beginning.

Reliability: CppGen is reliable in a way that it doesn't implement its own compiler but rather uses recognized tools used in computer world to make its own thing happen. There is little chance that building blocks that make CppGen will cause problems from the ground.

Main Goals of the Project:

-To create a world known website used by programmers, universities, students and all alike around the world.

-To contain no commercial purpose. Everything should be open source.

-To join hackathons, and be recognized.

-To make a website or a tool that has the capacity to be realized by authorities within the tech world.

-To receive some donation or funding if possible.

-To replace some of functionalities provided by commercial IDEs

-To be up to date.

-To create a functional and fast website without complexity.

-To be reliable.

-To be able to implement my own compiler(if at all) at some point rather than using already made tools.

-To feel that I didn't go to university for nothing.

-To know that it completely belongs to me with nobody got involved in the project besides

Before and After

How will this project fit into Senior Project?: Senior Project is a way for students to prove their undergraduate education means more than just a simple diploma, it requires students to build things which require combinations of different set of skills gained during their undergraduate program combined with their creativity. cppgen already has that. To build it, I needed to be more than a student, a programmer, a system administrator or a student.

Where am I?: Since many parts of this project had already been done before senior year even started, this project is not going to start from ground for the Senior Project. My website has already had its features. It has multi-threading execution, file upload, generation of couple files in a particular format and many more. It is in working condition but not really usable and practical yet. For detail FAQ page can be read at cppgen.com/faq.

Where will I be heading to?: I've already implemented many major goals I had had in mind. Along the way I have learned a lot as to what to expect, project management, possible failures, possible problems and differences between theory and practice. But that doesn't mean I don't have other plans. There are many more lessons to learn from not giving up and going on. Each step I will take will become another lesson.

My main intention is to make my existing website more usable, beautiful and famous. To make that I need to have a plan.

Things may/may not change: There are some building blocks that cppgen relies on but may change. One of them is gcc-xml program which is used for generating XML dumps of a C++ file. Although it provides project's base functionality, newer technologies should be sought and their compatibility should be checked with the existing project. Possible alternatives are:

GCC-XML: Cppgen is already built with gcc-xml. Source files uploaded to server are passed to gccxml, then temporary xml representations are read by a python library pygccxml. The problem with gcc-xml is it is old technology(supports C++98 standard) and doesn't follow newer standards. It is now succeeded by CastXML.

CastXML --> gcc-xml is replaced by castXML, it also supports C++11 standard. Problem here is that gccxml is fully compatible with python library pygccxml. I am not sure if CastXML is also compatible. I had contacted their implementors at some time ago. I will do so again to have some information.

Clang++ --> said to used to have XML dumping property. But not anymore. Clang still has AST(Abstract Syntax Tree) with itself. It just doesn't produce XML representation. If there is a way to parse Clang++ AST, then this is another option to use.

Doxygen --> This is a recent option I came across with. It has xml dumping property with support of C++11. However, it is said that it has few bugs as to C++11 syntax. Other problem is that I don't know if there exists a any possible way to parse xml output of the program. Otherwise I will have to implement my own, if you choose using doxygen.

UPDATE: I have tried to contact current developer of pygccxml. He had responded me long time ago but I am not sure if he will do the same this time. Also I have subscribed to castXML mailing list and will be posting some question in near future.

UPDATE 9/29/2015: I've talked with one of the professor as to get in idea about possible options. One possible option he offered was to parse AST (Abstract Syntax Tree) dumped by clang++ compiler. He said it could be parsed within the program (I didn't quite get what he meant by this), and could be used for the same function that gccxml is used for. What this means is that if I wanted to use clang++ compiler, then I would also have to implement my own pygccxml library which would be, of course, another project itself.

UPDATE 9/29/2015: Some comments on the internet suggest Doxygen. For now, I don't have an idea how doxygen could be used or would fit with this project.

Clang AST Introduction Video : https://www.youtube.com/watch?t=67&v=VqCkCDFLSsc

A video about castXML and integration with pygccxml : https://www.youtube.com/watch?v=O2lBgtaDdyk

Here is advantage of castxml over gccxml : http://permalink.gmane.org/gmane.comp.compilers.gccxml/730

UPDATE 09/30/2015: A response from castXML mailing list has just arrived. According to the response, although castxml is able to parse C++11 specification, its xml representation won't be C++11 but C++98 instead. Which would mean that header files submitted to my website using C++11 specification would not be rejected by the base program and be parsed but since output generated by castxml is not structured based on C++11 but rather C++98, automaticly generated code based on C++98 representation of a parsed c++11 parsed code would include no C++11 characterictics. So the question is, is it worth shifting the base technology? I will put a quote from the response,

On 09/30/2015 10:36 AM, Uğur Büyükdurak wrote:

> I would like to know whether it is possible or not to build and use castxml
> on windows machines.

Yes.

> I would also like to know whether it is compatible with pygccxml or not.

See here for pygccxml status with regard to castxml:

https://github.com/gccxml/pygccxml/issues/19

> It is also mentioned that a C++ compiler supporting c++11 syntax and
> clang/llvm compiler(SDK install tree built) are needed which I don't
> understand why installing process needs two different compilers at
> the same time. So basically why does castxml need clang and why does
> it need another C++ compiler.

CastXML needs the LLVM/Clang SDK in order to build.  While gccxml came
with its own patched copy of GCC, CastXML builds directly against an
externally-built LLVM/Clang.  Since Clang is implemented in C++11,
CastXML is also implemented in C++11.  Therefore one needs a C++11
capable compiler to build both of them.  The Clang compiler is not
executed or used to compile anything but it must be built to get its
SDK.

Once CastXML is built and installed then it works independently
from the original LLVM/Clang SDK installation.

> And one thing I forgot to ask, does castxml support C++11 specification?

CastXML does support parsing C++11 code but only the C++98 subset
of the interfaces will be included in the output (no APIs with
rvalue references for example).  One can pass -std=c++11 to castxml.

-Brad

Project Structure

1.1 Background

I have already listed project's background and its primary purpose in the previous field.

1.2 Project Objectives

There are couple objectives to get done. CastXml will be integrated into existing project. CastXml tests with pygccxml are already seen by me, for now they seem compatible to the extend I need them. There are problems when virtual functions and inheritance get involved. These problems need to be fixed. Other change to get done is to change structure of the source code. It needs to be redesigned based on design patterns to make it more modular. Structure of source code is not really flexible. Things that are piped to standard output and standard error should be directed to files in the filesystem. Later these files can be reached to print syntax errors from users' header files on html so that users can see their syntax errors and warning if there any. Setup scripts are needed to move the project among various machines. It is hard to manage every source file and third party libraries by hand. So there is a need for kind of source code management system that fits to the project's needs. User interface needs to be changed. Drag and Drop interface seems to be the best possible solution because it has the ability to provide ease of use for users. There will be also a commandline version of the program that will be used directly from the command line. For the command-line application there may have to be a need for a setup script. Source code will also be uploaded to github for public eyes can see it. To summurize:

1-)Castxml will be build from a source code in a linux machine (fedora probably), and will be integrated to the project.

2-)Fix problems with inheritance and virtual functions.

3-)Structure of Code

4-)Some of information on stdout and stderr should be reflected to user by means of HTML

5-)Various setup scripts depending on needs.

6-)Drop and drag user interface.

7-)Command line version of the program.

8-)Source code will be uploaded to Github for public trust and opinion along with help.

UPDATED:

--People I talk to have said it would be really useful to include separate program for generating makefiles.

9-)Separate section in the website that help create makefiles.

10-)Separate command line utility that helps people create makefiles.

11-)Adding compilation options from which people can choose.

12-)Performance Tests to see its worst case running time. For example O(n), O(logn).....

13-)Source code compilation options for those who don't have compilers installed in their computers.

1.3 Project Constraints

1-)Project shouldn't go out of scope.

2-)It is one-term project. Project for the second term will be a command-line chat application that does not maintain anything about its users actions

3-)There are not too much base technology available except writing my own compiler. Of course it is impossible for me to do so. (for now at least)

4-)Castxml will be used as base technology, other options has depreciated.

5-)Project is already written in Python, Python will be used extensively with Jinja2 Templating Engine and Flask Web Framework.

6-)Compilers to be used will be g++ and clang++.

1.4 Project Scope

An already installed linux system is needed. Full-featured Python IDE is crucial for big scale source code management. Pycharm had been used and will be used for the purpose. Fedora is the primary choice of linux distrubition due to the fact that it is one of the most popular distribution with a huge user community. Communication with castxml family is needed. It is important to subsricibe to their mailing list. A remote cluster is needed for deploying the final application and making it online. digitalocean.com is the primary choice of remote clusters because it allows complete installation of system from the beginning. That means it is easy to customize it based on different needs. There will probably a need for consulting about various aspects of Linux system. Internet is being considered to be primary source of information.

CppGen is only one term project. It is not going to cover second term. A commandline chat application will be the project for the second term.

UPDATE:

For compiling purposes Cmake is a crucial part of the project. There is no compiled version of CastXML in Linux repositories yet. For that reason, it is important to be able to compile CastXML by hand.

1.5 Project Assumptions

There is an assumption of castxml and pygccxml compatibility. There is also another really important assumption of Castxml's ability to parse C++11 specification. All the project will be built upon these two assumptions. A failure at one of two means possible failure for the project. Source code management and user interface are separate technologies that are not related to parsing C++ syntax. There are no assumptions made about those in terms of castxml and pygccxml. They can be implemented separately.

C++ Specifications Revised,

On Tue, Nov 3, 2015 at 6:50 PM, Ugur Buyukdurak <ubuyukd1@binghamton.edu> wrote: Hello Professor,

>I have a website that basically parses C++ code and produces another code. 
>I am actually using it as a senior project for this term. I am using gcc-xml as        
>the base technology to parse C++ code and probably I will shift to castxml as soon as possible since castXML is capable of parsing C++11 syntax. 
>But as I talked with its developers, its internal representation is C++ 98 specification.  
>I suppose it won't be able to print anything contained in C++11 specification 

>My question to you is how valid do you think C++ 98 specification will be in the future?

-It will still be used, but gradually become less important relative to C++11.

>Do you think there will be much change in the language in the following years?

-Hard to say. C++17 is the next major revision, but it's not clear whether or not it will really come out in 2017.

>do you think there is many difference between C++98, C++11 and C++14 specification in terms of syntax and functionality?

-C++98 to C++11 was a pretty big change.
-C++11 to C++14 was a small change.
-C++14 to C++17 is hard to say, since still in the future.

>Do you think that C++11 and C++14 redetermine syntax rules of existing C++98.

-Yes, at least some.

>Can anything that is capable of being parsed by C++98 also be parsed with newer C++ editions?

-As far as I know.

UPDATE November 04 2015

Michka Popoff <notifications@github.com> Oct 24 (11 days ago)

to gccxml/pygccxml I have great news. I just managed to finish the integration of castxml in pygccxml. The release 1.7.0 is ready to go. I tested it on OS X (with clang) and Linux (with clang and with gcc 4.8.2). All unit tests are passing.

@mamoll I'll probably make the new release in a few days. I need to update the changelog and the documentation. There is a new example which show what flags to use to setup the configuration for castxml. http://pygccxml.readthedocs.org/en/develop/examples/parsing/example.html If you have some time, can you give it a try with your project and with py++ ?

C++ 11 Improvements That may affect code generation compatibility

CppGen primarily concerns about .cpp generation from header files. Header files are used to declare different kind of classes. With C++11 there are new kind of syntax additions that may not be represented by castXML. Most important and most used of them are move constructors, move assignment operators, lambda functions, Rvalue references, constructors or destructors alike declared with default and delete keywords, changes in constructor declarations.

All of these can be parsed but not necessarily represented in castXml internal representation meaning that output should be identical with those of C++98. C++98 will most probably be used in the future for compatibility reasons. Output from CppGen in particular may be lack of move constructors and lambda functions since they will probably be the most by C++11 programmers.

CastXml is also not able to parse and represent template programs meaning that website probably will not be suitable for experienced programmers. (I just realized because I started template programming recently.) But it will nicely fit with students from novice and middle experience backgrounds.

Makefile generation utility is completely separate from C++ code generation so it won't concern whether a code is C++11, template or something else. All needed is files' names.

UPDATE: November 03 2015

Up to now, makefile utility of the website has been separated from code generation part of the website. Code generation had also included makefile generation with it. But there was no other way of generating makefiles without producing code. Now, users don't have to upload their files for makefile generation since makefiles only needs file names of source codes and header files. User can enter names of her/his files whether it is a header or a source code file, and server produces appropriate makefile for client. For submitting names through website interface, an update was needed where client type source code names. There is also set of choices that includes compiler options, so those who request makefiles from server can have it with the ability to provide extra compiling options. Most of the work command line version of makefile generation is also almost finished. People can download the command line version of makefile utility to use it independent of the website. It takes files names and compiler options as parameters and produces appropriate output.

In following two weeks integration of CastXML will be done with the existing project. It may take up to half week to one and a half depending on the integration of pygccxml and CastXML. As of now, my own opinion is it is not going to take more than one week because compatibility of CastXML and pygccxml is proven up to a certain point by their developers. After integrating CastXml with the website, it is important to test castXML's ability to parse C++11 syntax. That can also be done before any integration happens to have an idea about how much difference CastXML will create.

After CastXML integration, two most important updates will be drag and drop interface and also outputting syntax errors to the client. These were the two main issues I had struggled to understand. Especially drag and drop interface created by third party JavaScript library had acted really differently than I had expected. So I had decided not to use it at all. I still have difficulties putting inheritance on output and also having difficulties with output stderr to html page with strange behavior of Drag and Drop interface.

Strange behavior happening with Drag and drop interface is that during the upload of code from client side, server does not receive bunch of files at the same time. Server takes one file and tries to process it and upload another file and process it. Unfortunately this is not how the internal logic of website work. All files needs to be uploaded at the same time. So drag and drop interface and syntax fixes can take the most effort and time.

11/25/2015 Project Development Details

Project Development Details

For the complicated parts of the project to be done, it is necessary to have an Full-featured IDE is needed in a development machine. As stated previously on this page, Pycharm from Jetbrains is the best choice for the development purposes of this project. Project is solely based on Python and it is designed to work on Linux machine. Host computer's operating system had been chosen as Fedora 22 but since Fedora 22 had some bugs and made it harder to maintain the project, Ubuntu 14.04 LTS is chosen for development purposes. It allows programmer to start faster since it doesn't require too much configuration as opposed to Fedora. Both distributions' repositories don't serve castXml to their clients, only gccxml is supported. For this reason castxml is downloaded from its development page and build manually using LLVM/Clang SDK, a compiler supporting C++11, and cmake. Instructions for building castXML is somehow complicated therefore a mail is sent to mailing list of castXml by me to clarify building steps of castxml.

No Linux distribution comes with Oracle JRE or JDK installed on them, for this reason if Oracle Java is to be installed, it has to be installed manually. For my project, Pycharm required Oracle JRE to be installed on the machine(not exactly required, actually recommended, also open jdk could be used which is open implementation of Oracle Java). Oracle includes Java packages that is compatible with .rpm based packages whereas for .deb based distributions there is only binary files. Since it is cumbersome and hard to install Oracle Java manually, I have used following article from DigitalOcean.com to have some idea.

https://www.digitalocean.com/community/tutorials/how-to-manually-install-oracle-java-on-a-debian-or-ubuntu-vps/

Pycharm is also pre compiled program that doesn't need to be installed on a Linux machine. It is downloaded and run straight forward. There is only small process that creates symbolic links for pycharm to be run from command line. That's pretty much it.

Original source codes had to be downloaded from the server that hosts cppgen.com. I didn't have original source codes since I had formatted my computer. Server don't have password login option, so I had to obtain my private ssh key from my old hard drive. I used it to connect to the server and fetch source codes from it. Development is being done in a local machine that has Ubuntu LTS 14.04 installed on it.

There is one more thing to mention that is virtualenv property of python culture. Virtuelenv is a program that helps people create "virtual" python execution environments. With virtualenv one can install whatever version of python independently of each other allowing different projects to survive on the same system.

First time I had used these programs, the time I had spent to research and install these programs was not less than one week. I had done lots of research including reading manuals, watching videos, and trying to understand how these programs work under the hood. Since I am already experienced with these programs, it took no more than 4-5 five hours to step everything from the beginning. I've done no research but just used my existing information and experience to make the process faster.

Update: 11/26/2015 CastXML installation Process

First castxml source code is fetched from github page and cmake is used to create platform native compile tree (such as linux make). Due to vague instructions on the page and lack of experience with cmake, I was not able to compile directly from source code from github page. For this reason, I emailed castxml mailing list to find out what I can do about it. Came another vague explanation from there. But it was help in a way that it stated there is a prebuilt version of castxml present for Ubuntu. However, since the package is recently built, maintainer said that it may not be accessible through Ubuntu's package managere "apt-get". I found the repository and downloaded the program and tried to install it using dpkg of Debian systems. It didn't work.

Selecting previously unselected package castxml.

(Reading database ... 205934 files and directories currently installed.)

Preparing to unpack castxml_0.1+git20150807-1_amd64.deb ...

Unpacking castxml (0.1+git20150807-1) ...

dpkg: dependency problems prevent configuration of castxml:

castxml depends on libstdc++6 (>= 5.2); however:

Version of libstdc++6:amd64 on system is 4.8.4-2ubuntu1~14.04.

dpkg: error processing package castxml (--install):

dependency problems - leaving unconfigured

Processing triggers for man-db (2.6.7.1-1ubuntu1) ...

Errors were encountered while processing:

castxml

Answer from castXML mailing list,

Steve Robbins <steve@sumost.ca>
1:34 AM (19 hours ago)

to me, castxml 
Castxml has been packaged for debian/Ubuntu already. It's pretty recent so apt-get may not work for your distribution. 
But you can look at the source package to see in detail how to build it. 

I asked other questions also, but they didn't answer them,

I googled the problem around 1 and a half to find out what's going on. Since there is too few people that used castxml, it is hard to find answers from internet. One person said he came across with same problem with a different program. The reason he said if packages are built with a greater version than the OS you are using then it simply doesn't install. After learning that I checked what kind of compiled versions of castXml present. There are two versions present:

Castxml Compiled For Ubuntu Wily WereWolf 15.10 Castxml Compilde For Ubuntu Xenial Xerus 16.04 LTS

I had had Ubuntu 14.04.3 LTS installed on my host machine so I had to reinstall it. I wasn't able to find Xenial Xerus. Official Ubuntu download page has 15.10 on their page. So I grabbed 15.10 for the purposes of this project. Ubuntu 14.04.3 LTS is mentioned "recommended to most users", and it was the reason why I had first started with it(besides fedora 22 it had too many bugs for my computer). With Ubuntu 15.10 castxml works great.

One more thing to mention about Ubuntu 15.10 is that it has all new kind of software available from its package management system. Latest compilers from GNU project, clang and clang++ 3.7, gdb and latest version of valgrind are all present in repositories of Ubuntu 15.10.

Since I reinstalled the operating system from the beginning all of the steps explained had to be redone.

Update: 11/29/2015 CastXML Run Process

Now castxml is integrated into the project and works better than expected. There were too many errors at the beginning as to get it working. New version of pygccxml which is version 1.7.1 now fully supports castxml and even if there are some bugs, they don't interfere with the parts that I am interested in. I had edited original library when I first began the project to make it fit into my standards. Now the same thing has to be done once again for new version of pygccxml version 1.7.1. Since it is unnecessary time to make these changes to every upcoming version, it is better to write external wrapper that will handle the problem for me. I haven't tried running the program on C++11 syntax yet, but it will be the new goal. It works fine for the old syntax that everybody is familiar with. Below the conversation I had with castxml developer team:

Me:

I am also getting following error when I try to use castxml through pygccxml
RuntimeError: Error occured while running GCCXML: ['/usr/bin/castxml', '-I.', 'example.h', '--castxml-gccxml', '-o', '/tmp/tmpKsaXwW.xml']

Me:

Just to let you know, I had installed gccxml from Ubuntu repository after installing castxml. It wasn't normal gccxml but something more like wrapper gccxml     
around castxml. I have deleted it and castxml seems to work fine now. 
Do you know what's wrong with installing gccxml and castxml on the same machine? I am using Ubuntu 15.10

CastXml Team:

I wrote that wrapper so send bug reports my way. There should be nothing wrong with installing both packages.
It looks to me like you were invoking gccxml. If you want to use castxml then be sure to invoke castxml.

Me:

It may be the case, because pygccxml 1.7.1 finds the path to castxml or gccxml automatically. When they were both installed, pygccxml probably chose gccxml          
over castxml. But now the problem is RuntimeError: 
Error occured while running GCCXML: ['/usr/bin/castxml', '-I.', 'example.h', '--castxml-gccxml', '-o','/tmp/tmpKsaXwW.xml'] see the path, it points to      
'/usr/bin/castxml' so it doesn't seem like it is gccxml which is to be run. It doesn't make sense to me.

pygccxml current implementor:

pygccxml just tries to make a wild guess what to use.
If you have a mix of castxml/gccxml installed, or the wrapper version,
it may not be the one you want to use.

You need to set the path yourself by passing the xml_generator_path
argument to the configuration.

config = parser.xml_generator_configuration_t(
       xml_generator_path=generator_path,
       xml_generator=generator_name)

Note: RuntimeError: Error occured while running GCCXML just means
that gccxml (or castxml) crashed and did not write any xml file. This
is just the initial error you have mentioned.

Could you please provide a small c++ code example that makes
castxml fail ?