/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * @file MyroC-display-windows-opengl-img-handler.c * @brief Child Process for handling an image window on Windows 10 * This process is initiated in MyroC-display-windows-interface-manager.c * * OpenGL requires the window manager process is the primary thread * for controlling images, so a new process is needed for handling images. * on Mac OS X and Linux, this window can be spawned within a procedure * using a fork operation and pipes are used for communication between * the main program and the image-process manager * since fork is not available on Windows, * the window manager must be a separate program * the main program will set up communication, * the main program sets up input and output pipes * this window manager will use stdin/stdout for * I/O communication through the pipe * this version uses glutHideWindow when image windows time out * gluttHideWindow seems to work fine on Mac OS X and on some Linux machines * unfortunately, when run on amd graphics cards, glutHideWindow causes * window information to be lost, and subsequent calls to glutHowWindow fail * thus, this version of MyroC-display.c works fine on Mac OS X and * but not on Linux machines with various graphics cards * * Revision History * * Version 1.0 based on a C++ package by April O'Neill, David Cowden, * Dilan Ustek, Erik Opavsky, and Henry M. Walker * * Developers of the C package for Linux:s * Creators Version 2.0 (C functions for utilities,general,sensors,movement): * Spencer Liberto * Dilan Ustek * Jordan Yuan * Henry M. Walker * Contributors Version 2.2-2.3: (C functions for image processing) * Anita DeWitt * Jason Liu * Nick Knoebber * Vasilisa Bashlovkina * Revision for Version 2.4: (image row/column made to match matrix notation) * Henry M. Walker * * Revisions for Version 3.0 * Henry M. Walker * * C ported to Macintosh * Linux/Mac differences required for connections — otherwise same code * OpenGL used to display images, replacing ImageMagick * same [new] code used for both Linux and Macintosh * 1 process for robot control * 1 process needed for each titled window (not each image, as in 2.2-2.4) * Blocking options (negative duration parameter) * utilize separate thread timer * MyroC implementation files organized by user function as follows: * * Revisions for Version 3.1 * Henry M. Walker * * Picture struct and image functions revised to allow * 192 by 256 images from origial Fluke camera * 266 by 427 low-resolution images from Fluke 2 * (high-resolution (800 x 1280) too large for more than 2-4 * variables on run-time stack) * storage , retrieval, and display of any images up to 266 x 427 * * Revisions for Version 3.2 * Henry M. Walker * * Practical range of rBeep duration identified as <= 3.0014 seconds * Image display and processing resolves several matters and adds functionality * function rDisplayPicture completely rewritten * over time OpenGL rountines had encountered troubles * on Linux, specifically glutHideWindow() hid images, but could not be * restored, with difficulties depending on the graphics card * on Mac, High Sierra generated compile warnings and * restricted threads that could display images * function rDisplayImageProcessingErrors added * function rWaitTimedImageDisplay updated substantially * * Revisions for Version 3.3 * Henry M. Walker * * * * This program and all MyroC software is licensed under the Creative Commons * Attribution-Noncommercial-Share Alike 3.0 United States License. * Details available at http://creativecommons.org/licenses/by-nc-sa/3.0/us/ * ********************** implementation overview ************************* > indicates functions implemented in this file 0. LOW-LEVEL UTILITY (prototypes in MyroC-utilities.h implemented in MyroC-utilities.c) rSend rReceive rSetOpeningExchange rSetBluetoothEcho rSetEchoMode rCheckHardwareVersionSetCameraSize 1. GENERAL (mostly MyroC-general.c) 2. SENSOR (MyroC-sensors.c) rConnect (* MyroC-connect.c) a.Scribbler Sensors rDisconnect (* MyroC-connect.c) rGetLightsAll rSetConnection rGetLightTxt rFinishProcessing rGetIRAll rSetVolume rGetIRTxt rBeep rGetLine Beep2 rSetName b.Fluke Sensors rGetName rGetObstacleAll rSetForwardness rGetObstacleTxt rGetForwardness rGetBrightAll rSetLEDFront rGetBrightTxt rSetLEDBack rSetIRPower rGetBattery rGetStall Note: * MyroC-connect requires special knowledge of a Mac or Linux environment, yielding two versions: MyroC-connect-mac.c MyroC-connect-linux.c 3. MOVEMENT (MyroC-movement.c) 4. PICTURES (files as indicated) rTurnLeft rGetCameraSize (MyroC-camera.c) rTurnRight rTakePicture (MyroC-camera.c) rTurnSpeed > rDisplayPicture (MyroC-display.c) rForward > rDisplayImageProcessingErrors (MyroC-display.c) rFastForward > rWaitTimedImageDisplay(MyroC-display.c) rBackward rSavePicture (MyroC-image-file.c) rMotors rLoadPicture (MyroC-image-file.c) rStop rHardStop ************************************************************************ */ /* Overall processing constraints and plan Constraints Both OpenGL and ImageMagick on Linux and Mac OS X require a window to be controled as the primary thread of a process. Since a user may specify multiple windows (with different titles), the MyroC process must create a different process for each image window. OpenGL allows an window to be updated with a new image Thus, * the main MyroC process must be able to send a new message to the processing controling a window. * a pipe is needed from the main MyroC proces to a window process * once a window process displays an image the process must await a 'D' message to display a new image The GLUT package within OpenGL contains procedure glutHideWindow This works fine on Mac OS X This hides a window without trouble on Linux, but the corresponding glutPostRedisplay does not make a new window visible Thus, * MyroC will have to kill a process when a window is to be hidden, and a new window process created when a new image will be displayed at the same location with the same title. A window may disappear through either of two processes * If the window display is blocking, the window (and corresponding process) must go away when the duration interval is satisfied. * If the window display is non-blocking, * a timer thread can keep track of when the window should disappear * another user request may cause a new picture to be displayed, based on a new timing interval. */ /* Approach to the display of camera images * * approach outline based on two processes: * this [user] [main] program runs in one program * user/main program provides overall control of the robot, images, etc. * a child process provides OpenGL control for the display of images * * overall algorithm * utilize shared Picture struct variable * main process * provides general user interface * call to rDisplayPicture * if second/child process not running, * create pipe to send Picture images from main to child process * create child process * initialize graphics environment * establish shared Picture image as texture for image * establish new_image procedure as glutIdleFunc * if second process already running * first thread sends image through pipe to child process * redisplay process in child process * reads image * uses glTexSubImage2D to establish shared Picture image * uses glutPostRedisplay to force display of new image * image process (one process for each window) * create/initialize new image: procedure init_opengl_window * refresh image (required by openGL): procdure display * update image: procedure update_display * * synchronization protocol: all record keeping performed in main process * for new or updated image: * main process sends data to image process through socket * image process sends confirmation when done through socket */ /* OpenGL frameworks */ /* headers for any environment */ #include "MyroC.h" #include "MyroC-utilities.h" #include #include /* added to example for experimentation */ #include /* for strerror function */ #include /* for byte-level read/write */ #include #if defined(WIN32) || defined(_WIN32) || defined(__WIN32__) /* OpenGL and pipe headers for Windows 10 environment */ #include #include #include #include #include #else /* libraries for Linux and Mac */ #include /* mechanism for inter-thread communication */ #include /* needed to check if window process still exists */ #include /* for checking runtime error codes */ #include /* for killing display window, if necessary */ #include /* for determining if window process is running */ #include /* for threads, used in timing non-blocking images */ #ifdef __APPLE__ /* OpenGL headers for Mac OS X environment */ #include #else /* OpenGL headers for Linux environment */ #include #endif #endif /* ********************** Structure for timing display of images **************** */ typedef struct { double delay_time; int GLimageset_index; int image_seq_number; int thread_seq_number; } image_timing_data; /** * Struct for acting timing-thread information */ typedef struct { int thread_seq_number; pthread_t pthread; } timing_thread_info; /* ********** procedures needed in window process for openGL ********** */ /* initialize graphics environment */ void init_opengl_window (); //image process /* callback function called to redraw the scene/image being created */ void display (void); //image process /* callback function for use when CPU idle * used to handle processing other than window-based events * possible window-display updates: * either hiding current window * or preparing new image for display * receives input through pipe for displaying/updating/hiding images * format for data from pipe * byte 1: D display new image; * bytes 2-5: height of image * bytes 6-9: width of image * bytes 10-13: image sequence number * subsequent bytes: RGB pixel values as 1D array * byte 1: F current image display finished * bytes 2-5: image sequence number * */ void update_display (void); //image process /* ************* MyroC helper procedures ********************************* */ /** * take action to display image for the proper length of time */ void process_image_duration (double duration, int window_index); /** * timer to complete given openGL image after specified time */ void * display_timer_thread (void * delay_data); /* * @brief Create a new window to display a picture * @param pic RGB picture struct from Scribbler 2 camera * @param duration if duration > 0, operation is blocking * if duration <= 0, operation is non-blocking * for duration != 0, picture displayed for abs(duration) * seconds or until picture closed manually * if duration == 0, picture displayed until closed manually * @param window_index Index of existing window in image_array * white spaces will be replaced with underscores. * @param attempt_num count of how many attempts allowed before * giving up in creating new window * ( start >= 2, in case old window is being * killed when processing starts ) * @pre image_array[window_index] does not correspond * to an active window process */ void rDisplayPictureNewWindow (Picture * pic, double duration, const char * windowTitle, int window_index, int attempt_num); /* * @brief Update a picture in an existing window * @param pic RGB picture struct from Scribbler 2 camera * @param duration if duration > 0, operation is blocking * if duration <= 0, operation is non-blocking * for duration != 0, picture displayed for abs(duration) * seconds or until picture closed manually * if duration == 0, picture displayed until closed manually * @param window_index Index of existing window in image_array * white spaces will be replaced with underscores. * @pre 0 <= window_index < num_windows * image_array[window_index] corresponds to an active * window process */ void rDisplayPictureUpdate (Picture * pic, double duration, int window_index); /** * add thread sequence number, id, etc. to thread_timer array; */ void recordTimerThread (int thread_seq_num, pthread_t thread); /** * remove thread sequence number, id, etc. to thread_timer array; */ void deleteTimerThread (int thread_seq_num); /* ************* MyroC procedure prototypes **************************************** */ /** * @brief Display a picture in a new window * @param pic pointer to an RGB picture struct from Scribbler 2 camera * @param duration if duration > 0, operation is blocking \n * if duration <= 0, operation is non-blocking \n * for duration != 0, picture displayed for abs(duration) \n * seconds or until picture closed manually or * until the program terminates \n * if duration == 0, picture displayed until closed manually * @param windowTitle The title of the window that appears. * white spaces will be replaced with underscores. * @pre windowTitle is less than 100 characters. * @post image is displayed for the duration specified, \n * EXCEPT all display windows are closed * when the main program terminates. * @warning If images are displayed with a non-blocking option, and * if the user wants images to appear for a full duration, * use rWaitTimedImageDisplay or rFinishProcessing to block * processing until all image timers are finished. * Otherwise, program termination may close windows prematurely. */ void rDisplayPicture(Picture * pic, double duration, const char * windowTitle); /** @brief Turn on and off the printing of error messasge for image processing * @note Some operating systems (e.g., Mac OX S High Sierra) display * an error message whenevern a new process for image display starts * @post This function toggles this printing on and off * Initially, printing is off */ void rDisplayImageProcessingErrors (); /** * @brief Wait until all timed, non-blocking image window timers are complete * @post wait until all timed [non-blocking] images have closed \n * robot motion is unaffected by this function * * @warning images opened with duration 0.0 do not close until * images are updated or until the program terminates \n * all other image windows are closed by this function */ void rWaitTimedImageDisplay (); /* ******************* global variables for image handler ********************* */ /* Details of the picture for this image process */ #define max_image_width 427 #define max_image_height 266 int window_index; // index of this image within an overall grid char * window_title; // pointer to title string (unique for window) int height; // height of image object (192 for original fluke) int width; // width of image object (256 for original fluke) // OpenGL requires pixels in a 1D array in row-major order int max_pic_size = max_image_width*max_image_height; // maximum size based on Fluke 2 Pixel pixs [max_image_width*max_image_height]; // storage for 1D array of Pixels /* variables to allow logging of image handling */ int enableLogging = 0; // 1 - log operations while processing; 0 - no logging done char * logFileText = "image-handling-log.txt"; FILE * logFile; /* variables for openGL processes */ int image_seq_number = 0; // sequence number of current image int process_status; // current status of a display window process // values // 0: image process inactive, likely killed // 1: process active to display current image pthread_mutex_t lock; // mutex to prevent concurrent socket communication GLuint texture_objs [1]; // array for texture 1 object for openGL process int image_num; // sequence number of current/most recent image /* variablesfor keeping track of active threads */ int max_size_thread_array = 0; // maximum size of the current thread array int current_size_thread_array = 0; // number of elements actually containing thread info int current_thread_seq_num = 0; // a sequence number for identifying each thread int initial_thread_array_size = 10;// size of thread array when first initialized timing_thread_info * thread_info = NULL; // can't use malloc to initialize, so // actual initialization in first call to rDisplayPictureNewWindow /* variables for Windows pipes */ #define BUFSIZE 4096 CHAR chBuf[BUFSIZE]; DWORD dwRead, dwWritten; HANDLE hStdin, hStdout; /* *************************** Implementations **************************** */ /* **** Implementations: Image window functions ************************** */ /* initialize graphics environment */ void init_opengl_window () //image process { /* In graphics thread: * Declare initial window size, position, and display mode * (single buffer and RGBA). Open window with "hello" * in its title bar. Call initialization routines. * Register callback function to display graphics. * Enter main loop and process events. */ if (enableLogging) fprintf(logFile, " image process: starting creation of image window %s\n", window_title); /* set up command-line environment with no parameters */ int argc = 1; char * argv [1] = {"draw-texture"}; /* window initialization */ glutInit (&argc, argv); glutInitDisplayMode (GLUT_SINGLE | GLUT_RGB | GLUT_DEPTH); /* window initialization automatically defaults to pixels * non-pixel parameters not available * * parameters based on X, Y coordinates, not matrices */ glutInitWindowSize (width, height); /* position of window ultimately depends on operating system; * paramters here are suggestions to OS only * suggestions identify a grid of windows, 3-wide, * starting at the top of the monitor */ glutInitWindowPosition (470*((window_index)%3), 300*((window_index)/3)); /* display picture */ glutCreateWindow (window_title); /* select clearing (background) color * only floating-point parameters allowed */ glClearColor (0.0, 0.0, 0.0, 0.0); /* rasterize pixels directly, * rather than using interpolation between vertices of polygons */ glShadeModel (GL_FLAT); /* enable textures as background */ glEnable (GL_DEPTH_TEST); /* set pixel storage to byte alignment */ glPixelStorei (GL_UNPACK_ALIGNMENT, 1); /* name and create a texture object * this is accomplished by generating an int (actually GLubyte) index */ glGenTextures (1, texture_objs); /* the texture object will be a 2D array */ glBindTexture (GL_TEXTURE_2D, texture_objs[0]); /* specify that the texture will fill the image * other options: GL_REPEAT --- copy image as many times as needed * GL_CLAMP --- copy last pixel to end of image */ glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE); glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE); glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST); glTexParameteri (GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST); /* establish initial Picture as texture for image */ glTexImage2D (GL_TEXTURE_2D, 0, GL_RGB, width, height, 0, GL_RGB, GL_UNSIGNED_BYTE, (GLubyte *) pixs); /* set up callbacks */ glutDisplayFunc (display); glutIdleFunc (update_display); /* report to parent/scribbler process that initialization mostly done */ char ch = 'I'; if (enableLogging) fprintf (logFile , " image process: reporting window %s mostly set up\n", window_title); WriteFile(hStdout, &ch, 1, &dwWritten, NULL); /* start graphics event handler */ glutMainLoop (); } // init_opengl_window /* callback function called to redraw the scene/image being created */ void display (void) { /* clear all pixels */ glClear (GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT); glEnable (GL_TEXTURE_2D); glTexEnvf (GL_TEXTURE_ENV, GL_TEXTURE_ENV_MODE, GL_REPLACE); glBindTexture (GL_TEXTURE_2D, texture_objs[0]); /* initialize viewing values */ glMatrixMode (GL_PROJECTION); glLoadIdentity (); /* set coordinates to pixel numbers corresponding to * the size of the image */ gluOrtho2D (0, width, 0, height); /* invert the y axis, down is positive */ glScalef (1, -1, 1); /* move the origin from bottom left corner to the upper left corner */ glTranslatef (0, -height, 0); glMatrixMode (GL_MODELVIEW); /* map image to four corners of display area * note the order of the coordinates must describe successive * vertices of the surrounding window */ glBegin (GL_POLYGON); glTexCoord2f( 0.0, 0.0); glVertex2i ( 0.0, 0.0); glTexCoord2f( 1.0, 0.0); glVertex2i (width, 0); glTexCoord2f( 1.0, 1.0); glVertex2i (width, height); glTexCoord2f( 0.0, 1.0); glVertex2i ( 0.0, height); glEnd (); /* * start procesing buffered OpenGL routines */ glFlush (); glDisable (GL_TEXTURE_2D); } // display /* callback function for use when CPU idle * used to handle processing other than window-based events * possible window-display updates: * either killing current window * or preparing new image for display * receives input through pipe for displaying/updating/hiding images * format for data from pipe * byte 1: D display new image; * bytes 2-5: height of image * bytes 6-9: width of image * subsequent bytes: RGB pixel values as 1D array * byte 1: H hide current window * byte 1: 0 close current window/exit window process * */ void update_display (void) { /* window update not in progress */ /* read update request from pipe */ char command = 0; // set to 0 to anticipate handling process termination /* read window parameters */ BOOL bSuccess = ReadFile(hStdin, (char *) &command, 1, &dwRead, NULL); if (! bSuccess || dwRead == 0) { if (enableLogging) fprintf (logFile, " error in image-handling process: window command not communicated\n"); exit(1); } if (enableLogging) fprintf (logFile, " image process: update_display: received %c\n", command); switch (command) { case 'D': /* prepare next image for display */ { int new_width, new_height; // new block for declaring local variables /* need local block to declare height, width, seq_num */ bSuccess = ReadFile(hStdin, (char *) &new_width, 4, &dwRead, NULL); if (! bSuccess || dwRead == 0) { if (enableLogging) fprintf (logFile, " error in image-handling process: new image width not communicated\n"); exit(1); } bSuccess = ReadFile(hStdin, (char *) &new_height, 4, &dwRead, NULL); if (! bSuccess || dwRead == 0) { if (enableLogging) fprintf (logFile, " error in image-handling process: new image width not communicated\n"); exit(1); } /* check window dimensions have not changed */ if ((height != new_height) || (width != new_width)) { if (enableLogging) fprintf (logFile, "once window opened for a title, cannot change size\n"); return; } /* read pixels from pipe to get picture information */ for (int i = 0; i < width*height; i++) { bSuccess = ReadFile (hStdin, (char *) &pixs[i], 3, &dwRead, NULL); if (! bSuccess || dwRead == 0) { if (enableLogging) fprintf (logFile, " error in image-handling process: problem with Pixel data\n"); exit(1); } } /* next image in sequence being processed */ image_seq_number++; /* try to display next picture */ /* update existing texture with new Picture */ glTexSubImage2D (GL_TEXTURE_2D, 0, 0, 0, width, height, GL_RGB, GL_UNSIGNED_BYTE, (GLubyte *) pixs); if (enableLogging) fprintf (logFile, " image process: calling glutPostRedisplay\n"); /* report to parent/scribbler process that new image mostly done */ char ch = 'U'; // for "Updated" WriteFile(hStdout, &ch, 1, &dwWritten, NULL); if (enableLogging) fprintf (logFile, " image process: reporting window %s mostly set up\n", window_title); /* start graphics event handler */ glutShowWindow(); glutPostRedisplay (); } break; // end update display command; image completed == 1 to be processed yet case 'H': /* hide current image */ glutHideWindow (); /* report to parent/scribbler process that new image mostly done */ char ch = 'h'; // for "hidden" WriteFile(hStdout, &ch, 1, &dwWritten, NULL); break; // end hide image command case 0: /* null character received, perhaps due to closing of socket at end of processing exit image-window process */ if (enableLogging) { fprintf (logFile, " clean up: closing image window: %s\n", window_title); fclose (logFile); } exit (0); break; default: if (enableLogging) fprintf (logFile, "error (char %c/int %d) in handling image window update\n", command, command); exit (1); } // end switch statement options if (enableLogging) fprintf (logFile, " image process: command (char %c/int %d) completed\n", command, command); fflush (logFile); } // update_display /***** main program to establish OpenGL Image Handler process ******* this image-handler is based on these pre-conditions the window title is passed as a first argument I/O for this process goes through a previously-established pipe if error logging is enabled, error messages from this process go to logFile: image-handling-log.txt the first arguments received through the pipe are (in this order) the window index for this window the width and height of the image the pixel values for the initial image (RGB values in row-major order) */ int main ( int argc, char * argv []) { /* set up pipe I/O as stdin and stdout */ hStdout = GetStdHandle(STD_OUTPUT_HANDLE); hStdin = GetStdHandle(STD_INPUT_HANDLE); if ( (hStdout == INVALID_HANDLE_VALUE) || (hStdin == INVALID_HANDLE_VALUE) ) exit(1); if (argc >= 2) window_title = argv[1]; else window_title = "Untitled"; if (enableLogging) { logFile = fopen (logFileText, "a"); fprintf (logFile, "starting image handler process---"); fflush(logFile); } /* read window parameters */ BOOL bSuccess = ReadFile(hStdin, (char *) &window_index, 4, &dwRead, NULL); if (! bSuccess || dwRead == 0) { if (enableLogging) fprintf (logFile, "error in setting up image-handling process: window index %d/bSuccess %d - not communicated\n", window_index, bSuccess); exit(1); } if (enableLogging) fprintf (logFile, "window index: %d\n", window_index); bSuccess = ReadFile(hStdin, (char *) &width, 4, &dwRead, NULL); if (! bSuccess || dwRead == 0) { if (enableLogging) fprintf (logFile, " error in setting up image-handling process: image width not communicated\n"); exit(1); } bSuccess = ReadFile(hStdin, (char *) &height, 4, &dwRead, NULL); if (! bSuccess || dwRead == 0) { if (enableLogging) fprintf (logFile, " error in setting up image-handling process: image width not communicated\n"); exit(1); } /* read pixels for initial image */ for (int i = 0; i < width*height; i++) { bSuccess = ReadFile (hStdin, (char *) &pixs[i], 3, &dwRead, NULL); if (! bSuccess || dwRead == 0) { if (enableLogging) fprintf (logFile, " error in setting up image-handling process: problem with Pixel data\n"); exit(1); } } /* create thread_array for display timers, when function first called */ if (thread_info == NULL) { thread_info = (timing_thread_info*) malloc(initial_thread_array_size*sizeof (timing_thread_info)); max_size_thread_array = initial_thread_array_size; } /* create new openGL window process and prepare for image processing */ if (enableLogging) fprintf (logFile, " creating new image window %s\n", window_title); /* when preparation done, image will be visible */ process_status = 1; /* utilize OpenGL steps to create a new image window */ init_opengl_window (); /* window creation process does not return, but include return 0 for compiler checking */ return 0; }