Kopernik: Difference between revisions
From CS486wiki
Jump to navigationJump to search
Content deleted Content added
No edit summary (change visibility) |
|||
| (59 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
== '''''Week One''''' == |
|||
__TOC__ |
|||
== Introduction to Kopernik Observatory== |
|||
Kopernik Observatory & Science Center is a non-profit organization which is opened in 1974. Located atop a 1720 foot hill in Vestal, NY, the Observatory has been one of the best equipped public observatories in the Northeast United States for nearly 40 years. It provides educational program for students and their families. |
|||
== Information about Previous Project == |
|||
=== Relational Database === |
|||
In the previous project, previous team created a new database and combined four databases into one and implemented ten tables inside of the database. As it is mentioned in their final report, it is relation-based SQL file and it is implemented in MySQL environment. The previous project team only worked on the Kopernik Talent Search database which has 4894 rows (including some duplications). We contacted with a team member of the previous project Nagihan Cay and learned that they used Oracle MySQL Workbench Tool to extract the data from excel tables to import them into MySQL. Since the table names and its attributes might be changed in the future, the document does not consists of their information. |
|||
[[File:Kopernik_ER.jpg|frame|none|alt=Alt text|Entity-Relationship Model]] |
|||
=== Web Application === |
|||
By reading the previous project’s user manual, we found out that they designed some new General User Interface (GUI) and also they made some improvements on the web pages which are created by the project team before them. This improvements involve more input fields and student search boxes. They created several user-friendly web pages and explained these web pages purposes in their final report. A screenshot of an example for a web page (Figure 2[2]) can be found in the appendices part. |
|||
== Information about Current Project == |
|||
=== Objective === |
|||
Kopernik Observatory gives tens of educational programs to hundreds of students. The information about the students, their families, class, and memberships requires a big database which cannot be handled manually. Their old database is implemented in FileMaker Pro7. Although it is an effective technology for small companies, Kopernik has lots of data to store. The high amount of data makes the FileMaker Pro7 very inefficient. Not the database structure itself, also the user interface of the FileMaker is very complicated. It has lots of textboxes and unordered lists which makes users to confuse about what they are doing, especially the unexperienced users. |
|||
'''Title:''' Weekly Updates on Kopernik |
|||
Kopernik Science & Observatory Center Database and Web Application Project is created by previous project team to improve the Kopernik’s current database system and its interface. To achieve this, the previous team started implementing a new relational database structure and a web application. Our project has three stages: |
|||
'''Date:''' 02/06/2022 |
|||
#Database Stage – In this stage, we will start examining the database structure which is created by the previous project team, then we will continue from where they left off. Our main priority will be the reconstruct of the current database system. |
|||
#Web Application Page Stage – In this stage, we will check the previous project team’s web pages and decided the adequacy of the current web pages. If there will be a need for more pages, we will create them. |
|||
#Miscellaneous Stage – In the miscellaneous stage we will try to fulfill the extra specifications. This stage can rather be done or left out according to our schedule. If we can finish the first two stages, we will focus on the more “unimportant” functions or modules such as “Spell checking”, “go back buttons” etc. |
|||
'''Subject:''' Getting Started |
|||
=== Project Scope === |
|||
The project specifications are defined by our customer Kopernik Science & Observatory Center’s Executive Director, Drew Deskur and our employer Professor Richard Steflik. This specifications will be used as a definition of this project. The main requests of the customer are: |
|||
Integration of all databases |
|||
'''Attendance:''' Baris Izar, oguz Kircan, Ata Akin, Professor Jones, Andrew Deskur |
|||
==== Combination of Databases ==== |
|||
The current system has three different databases; Student, membership and scheduling. When users want to change something they also need to change the other two database entries manually. For example: If a user wants to add two siblings and their parent’s information to database, he/she must have to add parent information for both children. This method not only wastes time but also it is very open to false data entry. Also by adding parent information two times for both children, user makes duplication of data without his/her purpose. Also, due to duplications, when a user wants to change an information on the database, he/she must update the database for twice. In a case where user changes one of the values and forgets to change the other one, data corruption would begin in the database. This would make a critical damage to the organization if this happens to the tables where the premium members’ information hold. |
|||
==== Creation of the User interface ==== |
|||
User interface of the FileMaker Pro7 is very simple and elegant for small companies with small databases but Kopernik has huge amount of data. This created a very complex user interface which involves lots of database name labels (user simply clicks these labels to view the data held inside). These labels are listed in a very scattered way with lots of different colors. Also the forms -which are used to add new data or modify the current ones- look very complex with absurd textboxes. Previous project team created a new, simple user interface with web pages. If more needed, we will continue to add new web pages. |
|||
'''Meeting Agenda''' |
|||
---- |
|||
In the miscellaneous stage, customer has some extra requests. These requests are: |
|||
*A function that makes a statistical analysis and creates a report about the classes, students, members etc. |
|||
*Printing a report of payment history for a specified class. |
|||
*Rename of the database table names which are showed in the user interface. For example: Instead of “SSC11” program should show “Summer Space Camp 2011” |
|||
*Printing all classes that one specific student takes. |
|||
*Printing a report which contains specific student information. |
|||
*Customer wants to see the last modification date and time of entries in order to see the members’ membership renewal time. |
|||
*Auto-fill function on text boxes. |
|||
*Field Check for preventing false or redundant entries. For example: State field should only take two characters. |
|||
*Customer wants a warning box which warns users if a field is left blank on a new entry. |
|||
*Customer wants confirmation box which asks a question such as “Are you sure” while saving a new entry or editing an old one. |
|||
As mentioned before, the miscellaneous stage consists the optional requests from the customer. These specifications which are mentioned above under the miscellaneous stage, do not have any critical role in the program. |
|||
- Satisfying necessary paperwork before starting the Kopernik Project. |
|||
== Week 1 & 2 == |
|||
Installation of previous project completed with no errors. Tested every web page simultaneously with the database, in order to check every single page is working correctly. Found couple mistakes and missing parts in previous project. For example: As we understand. previous project team changed their original code to test their functions or forgot to increment some counters which caused a database corruption. In this case, if admin wanted to create a new student with two parents, it is created with 7 parents(kinda funny). |
|||
'''Current Status''' |
|||
Started to add form validations on some of the pages, such as text requirements, email confirmation, etc. |
|||
- Prof. Jones shared the information about the project with us in our first meeting. |
|||
== Week 3 == |
|||
Started to work on pages that are not completely done. For example previous project cannot add new students correctly. There was some complications about students parent and family information. Changed that and now add new student page working correctly. |
|||
- Task distribution was made, including who worked in which part of the project. |
|||
Our client wants to add new students to existing families. There was nothing about that in the previous project. Added necessary pages in the add new student page and modified database according to that. |
|||
- Ata Akin will continue to look at the meteor m2 weather satellite that he has been working on in the previous period. |
|||
== Week 4 == |
|||
Continue working on adding new student to existing families page. Started doing an edit page in order to edit family information. |
|||
- Baris Izar will display the images obtained from the satellites that will work on the FTP server on its website. |
|||
== Week 5 == |
|||
Added a search feature which works with "Family Last Name". Completed the page which enables user(s) to add a new student to an existing family. User(s) can now see the family information without making any changes. |
|||
- Oguz Kircan will provide support to both sides and monitor the progress of the project. |
|||
Form validations are implemented and tested for "Add New Student" page. With the validation function, user is warned if he/she enters a non-valid entry(ex: using a letter in a zip code input area.) On a successful attempt, the page returns a warning which states "Record is added!" |
|||
Had a meeting with Drew Deskur to discuss about our progress and new features. |
|||
'''Accomplishments''' |
|||
- Set up a time for weekly meetings with Prof. Jones, Andrew Deskur, Ata Akin , Oguz Kircan and Baris Izar. |
|||
- We (Oğuz and Barış) started to study Selenium and Wordpress. |
|||
- Ata informed us about the project more in depth. |
|||
== '''''Week Two''''' == |
|||
'''Title:''' Meeting with Andrew Deskur |
|||
'''Date:''' 02/13/2022 |
|||
'''Subject:''' Kopernik Project Requirements |
|||
'''Attendance:''' Oguz Kircan, Baris Izar, Ata Akin, Professor Jones, Andrew Deskur |
|||
'''Meeting Agenda''' |
|||
- We showed what we did in the past week. |
|||
- Ata was able to execute code for capturing the images. |
|||
- Oguz and Baris found the tutorial about word press and the raspberry pi to better understand and study. |
|||
'''Current Status''' |
|||
- After open the weather folder after you got in when you called predict software it asks for Ground station location I already entered the values when you did that once you do not need to do it again. Which helps to predict software to understand your location and when the satellite passes close to ground station the program will know. |
|||
'''Accomplishments''' |
|||
- |
|||
== '''''Week Three''''' == |
|||
'''Title:''' Meeting with Andrew Deskur |
|||
'''Date:''' 02/20/2022 |
|||
'''Subject:''' Kopernik Project Requirements |
|||
'''Attendance:''' Oguz Kircan, Baris Izar, Ata Akin, Professor Jones, Andrew Deskur |
|||
'''Accomplishments:''' |
|||
- Baris connected to the VNC viewer without a problem. |
|||
- find_images() method inside the ftp_allfiles.py, was not functioning. Now, it's debugged and working. |
|||
- ftp_allfiles.py runs without giving any errors. It creates a ftp object and connects to the host through it. |
|||
- The data transfer is also will be through that object. It is also possible to change the directory |
|||
'''Problems''' |
|||
- Even though the code seems working, we are not sure how to test if ftp_allfiles.py really uploades the images. |
|||
'''To Do''' |
|||
- Test if ftp_allfiles.py successfully handles the file transfer. |
|||
== '''''Week Five''''' == |
|||
'''Accomplishments:''' |
|||
- |
|||
'''Problems:''' |
|||
New NOAA GOES server is still freezing and too laggy to use. We are trying to understand what's causing it. |
|||
Things we had checked so far: |
|||
* Running out of memory |
|||
There's still 11 gb available memory. So, this is not the problem. |
|||
* Operating System that VNC is installed on |
|||
It freezes on both Windows and MAC. So, it is not likely. |
|||
* Missing updates |
|||
There are two packages that need upgrading. They may be the cause. |
|||
* Hit:1 http://archive.raspberrypi.org/debian bullseye InRelease |
|||
* Hit:2 http://raspbian.raspberrypi.org/raspbian bullseye InRelease |
|||
We did not update them yet. |
|||
* CPU Usage |
|||
When we checked the CPU usage of the softwares', we saw |
|||
that their CPU usage mostly differs between 15-40 percent. |
|||
At the beginning, we thought this would be the problem. Yet, |
|||
it was the same with the NOAA APT server and it seems |
|||
working properly. |
|||
* Softwares |
|||
Problem might be caused by a software, which is installed on New GOES and not installed on Kopernik NOAA APT. There are only two softwares meeting the criteria: |
|||
* ImageMagick |
|||
Deleting ImageMagick did not effect the freezing problem. |
|||
* GNU Radio Companion |
|||
It's still trying to remove the packages. |
|||
'''TO DO:''' |
|||
* Find what's causing the abnormal lag on NOAA GOES server and solve it. |
|||
* Try to implement the new Meteor-M 2 satellite imagery capture setup. |
|||
== '''''Week Seven''''' == |
|||
'''Accomplishments:''' |
|||
* Raspberry NOAA V2's installation is complete and it is ready to use. |
|||
* We were getting an error saying that "No package matching "libjpeg9" is available." This took the most of our time. In order to solve that, we comment out the dependencies including libjpeg9; and install libjpeg62-turbo manually instead. |
|||
* We did the same for the package "predict" as well. |
|||
* During the installation, we follow the following link: https://github.com/jekhokie/raspberry-noaa-v2 |
|||
* To overcome the "No package matching ..... is available error", we follow the following link: https://github.com/jekhokie/raspberry-noaa-v2/issues/515 |
|||
'''TO DO:''' |
|||
* We will observe the pi to see is it working or not or is it freezing. |
|||
== '''''Week Eight - 04-04''''' == |
|||
'''Accomplishments:''' |
|||
* We got the antenna from Drew and set it up. |
|||
* We updated the system according to the Drew's email. |
|||
The station information (predict): |
|||
Call Sign: K2ZRO |
|||
Latitude : 42°00.14' If you need decimal degrees use: 42.003889 |
|||
Longitude: 76°02.01' (76.033611 or - 76.033611 |
|||
Altitude : 521 meters |
|||
Settings.yml: |
|||
Latitude : 42.003889 |
|||
Longitude : -76.033611 |
|||
Altitude : 521 |
|||
Timezone offset: -4 |
|||
Ntp server: time-a-g.nist.gov |
|||
meteor_receiver: 'rtl_fm' |
|||
noaa_receiver: 'rtl_fm' |
|||
'''TO DO:''' |
|||
* We will check where the images are saved and see if the new NOAA V2 is working. |
|||
* We will check the after installation part of the jekhokie's github repository and see if we could find any other useful feature to implement. |
|||
== '''''Week Ten- 17-04''''' == |
|||
'''Accomplishments:''' |
|||
* We installed "fbi" to view the images on Raspberry Pi Lite and now, we are able to view them. More info can be found on the following link. |
|||
https://www.raspberrypi-spy.co.uk/2017/02/how-to-display-images-on-raspbian-command-line-with-fbi/ |
|||
* We connected the Raspberry Pi to ssh, using PuTTy, so that we could use multiple terminals to make various tests. |
|||
* We test the reception, which will validate if the antenna, reception line, reception hardware and software is working correctly. We should've heard a voice, but there wasn't any. |
|||
* We checked that if the antenna is connected to the rtl fm radio, using a headphone. It seems okay. |
|||
* We checked the schedule with "crontab - l". All seems fine. |
|||
* If the photo is captured in daylight, our sum_min_elevation variable must be belove the sun angle. In the last capture, the sun angle was 25, and our sum_min_elevation is 6. So, this seems fine as well. |
|||
'''Problems:''' |
|||
* All the satellite images are completely red. |
|||
* We are trying to add the functionality to pushing all captured, processed images to a destination email address. We could not manage to do that yet. |
|||
'''TO DO:''' |
|||
* We will figure out what's causing the images to become red. |
|||
* We will add the functionality to push the captured images as email. |
|||
* We will make further research for the reception. |
|||
*We will check the noise level. The process will take five hours and produce a heatmap waterfall image. |
|||
== '''''Week Eleven - 24-04''''' == |
|||
'''Accomplishments:''' |
|||
* Added a new file to the NOAA APT |
|||
* New python file wpbot2 is doing to the same thing as wpbot.py but the only difference is it loads images to the NOAA15 folder instead of the weather folder, and also deletes the previous files. |
|||
* We setup the new Raspberry Pi Lite at Kopernik Observatory. |
|||
'''TO DO:''' |
|||
* We'll check if the images are still red. |
|||
== '''''Week Twelve - 02-05''''' == |
|||
'''Accomplished''' |
|||
* We managed to display the images on PC, using the web panel, which contains the images from passes, and some information regarding the schedules and the satellites' locations. The meteor images are seem pretty clear. |
|||
* In order to use the web panel, while you are connected the same internet connection with pi and with your PC, you need to connect pi's local ip adress. In this case we connected to " https://192.168.1.240/" which directs us to web panel. You can change the web panel settings from raspberry-noaa-v2/config/settings.yml". |
|||
* We managed to get the images from pi to our PC. This process can be done in two ways. One does not require any software, yet you can only send one image at a time to your PC. More info can be found on the following video: |
|||
https://www.youtube.com/watch?v=rsov0FEhZtk&ab_channel=AbhayKant |
|||
* The other option to transfer files between pi and Pc, is an application called "WinSCP". We used this one to trasnfer all the previous images to our local PC. The software can be downloaded from the following link: |
|||
https://winscp.net/eng/index.php |
|||
* We managed to get the pictures according to the noaa satellite numbers and upload them to the website with their noaa satellite numbers. (Baris' part) |
|||
'''Problems''' |
|||
* Only the meteor images are there. There are no images from any NOAA satellite, only the spectograms. |
|||
'''To Do:''' |
|||
* Understand why there are no NOAA satellite images. |
|||
* We'll figure out how to send the images to the website. |
|||
Latest revision as of 14:57, 4 May 2022
Week One
Title: Weekly Updates on Kopernik
Date: 02/06/2022
Subject: Getting Started
Attendance: Baris Izar, oguz Kircan, Ata Akin, Professor Jones, Andrew Deskur
Meeting Agenda
- Satisfying necessary paperwork before starting the Kopernik Project.
Current Status
- Prof. Jones shared the information about the project with us in our first meeting.
- Task distribution was made, including who worked in which part of the project.
- Ata Akin will continue to look at the meteor m2 weather satellite that he has been working on in the previous period.
- Baris Izar will display the images obtained from the satellites that will work on the FTP server on its website.
- Oguz Kircan will provide support to both sides and monitor the progress of the project.
Accomplishments
- Set up a time for weekly meetings with Prof. Jones, Andrew Deskur, Ata Akin , Oguz Kircan and Baris Izar.
- We (Oğuz and Barış) started to study Selenium and Wordpress.
- Ata informed us about the project more in depth.
Week Two
Title: Meeting with Andrew Deskur
Date: 02/13/2022
Subject: Kopernik Project Requirements
Attendance: Oguz Kircan, Baris Izar, Ata Akin, Professor Jones, Andrew Deskur
Meeting Agenda
- We showed what we did in the past week.
- Ata was able to execute code for capturing the images.
- Oguz and Baris found the tutorial about word press and the raspberry pi to better understand and study.
Current Status
- After open the weather folder after you got in when you called predict software it asks for Ground station location I already entered the values when you did that once you do not need to do it again. Which helps to predict software to understand your location and when the satellite passes close to ground station the program will know.
Accomplishments
-
Week Three
Title: Meeting with Andrew Deskur
Date: 02/20/2022
Subject: Kopernik Project Requirements
Attendance: Oguz Kircan, Baris Izar, Ata Akin, Professor Jones, Andrew Deskur
Accomplishments:
- Baris connected to the VNC viewer without a problem.
- find_images() method inside the ftp_allfiles.py, was not functioning. Now, it's debugged and working.
- ftp_allfiles.py runs without giving any errors. It creates a ftp object and connects to the host through it.
- The data transfer is also will be through that object. It is also possible to change the directory
Problems
- Even though the code seems working, we are not sure how to test if ftp_allfiles.py really uploades the images.
To Do
- Test if ftp_allfiles.py successfully handles the file transfer.
Week Five
Accomplishments:
-
Problems:
New NOAA GOES server is still freezing and too laggy to use. We are trying to understand what's causing it.
Things we had checked so far:
- Running out of memory
There's still 11 gb available memory. So, this is not the problem.
- Operating System that VNC is installed on
It freezes on both Windows and MAC. So, it is not likely.
- Missing updates
There are two packages that need upgrading. They may be the cause.
* Hit:1 http://archive.raspberrypi.org/debian bullseye InRelease * Hit:2 http://raspbian.raspberrypi.org/raspbian bullseye InRelease
We did not update them yet.
- CPU Usage
When we checked the CPU usage of the softwares', we saw that their CPU usage mostly differs between 15-40 percent. At the beginning, we thought this would be the problem. Yet, it was the same with the NOAA APT server and it seems working properly.
- Softwares
Problem might be caused by a software, which is installed on New GOES and not installed on Kopernik NOAA APT. There are only two softwares meeting the criteria:
* ImageMagick
Deleting ImageMagick did not effect the freezing problem.
* GNU Radio Companion
It's still trying to remove the packages.
TO DO:
- Find what's causing the abnormal lag on NOAA GOES server and solve it.
- Try to implement the new Meteor-M 2 satellite imagery capture setup.
Week Seven
Accomplishments:
- Raspberry NOAA V2's installation is complete and it is ready to use.
- We were getting an error saying that "No package matching "libjpeg9" is available." This took the most of our time. In order to solve that, we comment out the dependencies including libjpeg9; and install libjpeg62-turbo manually instead.
- We did the same for the package "predict" as well.
- During the installation, we follow the following link: https://github.com/jekhokie/raspberry-noaa-v2
- To overcome the "No package matching ..... is available error", we follow the following link: https://github.com/jekhokie/raspberry-noaa-v2/issues/515
TO DO:
- We will observe the pi to see is it working or not or is it freezing.
Week Eight - 04-04
Accomplishments:
- We got the antenna from Drew and set it up.
- We updated the system according to the Drew's email.
The station information (predict):
Call Sign: K2ZRO
Latitude : 42°00.14' If you need decimal degrees use: 42.003889
Longitude: 76°02.01' (76.033611 or - 76.033611
Altitude : 521 meters
Settings.yml:
Latitude : 42.003889
Longitude : -76.033611
Altitude : 521
Timezone offset: -4
Ntp server: time-a-g.nist.gov
meteor_receiver: 'rtl_fm'
noaa_receiver: 'rtl_fm'
TO DO:
- We will check where the images are saved and see if the new NOAA V2 is working.
- We will check the after installation part of the jekhokie's github repository and see if we could find any other useful feature to implement.
Week Ten- 17-04
Accomplishments:
- We installed "fbi" to view the images on Raspberry Pi Lite and now, we are able to view them. More info can be found on the following link.
https://www.raspberrypi-spy.co.uk/2017/02/how-to-display-images-on-raspbian-command-line-with-fbi/
- We connected the Raspberry Pi to ssh, using PuTTy, so that we could use multiple terminals to make various tests.
- We test the reception, which will validate if the antenna, reception line, reception hardware and software is working correctly. We should've heard a voice, but there wasn't any.
- We checked that if the antenna is connected to the rtl fm radio, using a headphone. It seems okay.
- We checked the schedule with "crontab - l". All seems fine.
- If the photo is captured in daylight, our sum_min_elevation variable must be belove the sun angle. In the last capture, the sun angle was 25, and our sum_min_elevation is 6. So, this seems fine as well.
Problems:
- All the satellite images are completely red.
- We are trying to add the functionality to pushing all captured, processed images to a destination email address. We could not manage to do that yet.
TO DO:
- We will figure out what's causing the images to become red.
- We will add the functionality to push the captured images as email.
- We will make further research for the reception.
- We will check the noise level. The process will take five hours and produce a heatmap waterfall image.
Week Eleven - 24-04
Accomplishments:
- Added a new file to the NOAA APT
- New python file wpbot2 is doing to the same thing as wpbot.py but the only difference is it loads images to the NOAA15 folder instead of the weather folder, and also deletes the previous files.
- We setup the new Raspberry Pi Lite at Kopernik Observatory.
TO DO:
- We'll check if the images are still red.
Week Twelve - 02-05
Accomplished
- We managed to display the images on PC, using the web panel, which contains the images from passes, and some information regarding the schedules and the satellites' locations. The meteor images are seem pretty clear.
- In order to use the web panel, while you are connected the same internet connection with pi and with your PC, you need to connect pi's local ip adress. In this case we connected to " https://192.168.1.240/" which directs us to web panel. You can change the web panel settings from raspberry-noaa-v2/config/settings.yml".
- We managed to get the images from pi to our PC. This process can be done in two ways. One does not require any software, yet you can only send one image at a time to your PC. More info can be found on the following video:
https://www.youtube.com/watch?v=rsov0FEhZtk&ab_channel=AbhayKant
- The other option to transfer files between pi and Pc, is an application called "WinSCP". We used this one to trasnfer all the previous images to our local PC. The software can be downloaded from the following link:
https://winscp.net/eng/index.php
- We managed to get the pictures according to the noaa satellite numbers and upload them to the website with their noaa satellite numbers. (Baris' part)
Problems
- Only the meteor images are there. There are no images from any NOAA satellite, only the spectograms.
To Do:
- Understand why there are no NOAA satellite images.
- We'll figure out how to send the images to the website.
