TABLE OF CONTENTS

RPN_COMM/io_pe management routines for IO PE sets [ package ]

[ Top ] [ package ]

FUNCTION

   Create, free, manage sets of processes used to perform IO (or other) operations
   in the RPN_COMM library (version 4.5.16 and above)

AUTHOR

   M.Valin Recherche en Prevision Numerique 2015

DESCRIPTION

  user callable routines / functions (output/inout arguments in UPPER CASE)
                 CREATE / FREE
   - setno      = RPN_COMM_create_io_set(npes,method)
   - call         RPN_COMM_make_io_pe_list(X,Y,npes,penx,peny,method)
   - setno      = RPN_COMM_free_io_set(setno)
                  INFORMATION
   - .t /.f.    = RPN_COMM_is_valid_io_setno(setno)
   - ordinal_s  = RPN_COMM_is_io_pe(setno)
   - ordinal_g  = RPN_COMM_io_pe_gridid(setno,n)
   - comm       = RPN_COMM_io_pe_comm(setno)
   - npes       = RPN_COMM_io_pe_size(setno)
   - grps       = RPN_COMM_io_pe_groups(setno)
   - list(:)    = RPN_COMM_io_pe_idlist(setno)
   - coord(:,:) = RPN_COMM_io_pe_coord(setno)
                    OPERATE
   - ivalue     = RPN_COMM_io_pe_callback(setno,callback,argv)
   - call         RPN_COMM_io_pe_bcast(BUFFER,count,datatype,root,setno,IERR)
                   VALIDATION
   - 0/-1       = RPN_COMM_check_ioset(setno, x ,y, npes, penx, peny, peme, diag)
   - logical    = RPN_COMM_io_pe_valid_set(X,Y,npes,penx,peny,diag,method)

     setno      set number, assigned at IO PE set creation time by RPN_COMM_create_io_set
     method     PE spread method, integer, must be 0 for now
     peme       ordinal in "GRID" of this PE
     diag       if .true. and peme == 0, print diagnostic messages
     npes       number of PEs in IO set
     grps       number of "groups" ni thi IO PE set
     comm       MPI communicator for this IO PE set (integer)
     ordinal_s  ordinal in IO PE set communicator  (origin 0)
     ordinal_g  ordinal in "GRID"  (origin 0)
     list       one dimensional integer array, list(I) is the ordinal in the "GRID"
                communicator of the Ith IO PE in set. (size of list is npes)  (origin 0)
     coord      two dimensional integer array (npes,2) containing the X and Y coordinates
                in the "GRID" of the IO PEs in the set.   (origin 0)
                X coordinates in coord(:,1), Y coordinates in coord(:,2)
     argv       "blind" argument of type(C_PTR) that will be passed to callback
     ivalue     integer return value of callback function
     callback   the name of a user supplied function, passed to RPN_COMM_io_pe_callback.
                RPN_COMM_io_pe_callback MUST be called on ALL members of the IO PE set
                but may also be called by non members of the IO set. the callback function
                will only be called on the members of the IO set. the return value of 
                RPN_COMM_io_pe_callback will be 0 on non members and the return value of 
                callback on the members of the IO set. the call to callback is done as
                ivalue = callback(argv,setno,comm,pe_me,x,y,npes)
                pe_me is the ordinal of the PE in the IO PE set communicator (NOT "GRID")

Notes

     include 'RPN_COMM.inc'
     is necessary to get the correct interface description

     the normal sequence of operations is 
     1 - RPN_COMM_create_io_set
     2 - get info, do operations on set "setno"
     3 - RPN_COMM_free_io_set

     for now a maximum of 16 IO PE sets may be defined at any point in time
     (when an IO set has been freed, it does not count anymore)

     RPN_COMM_make_io_pe_list and RPN_COMM_check_ioset are really internal routines,
     users should rather call RPN_COMM_io_pe_valid_set that is intended for public use
     and better documented

     FOR MODEL DEVELOPERS:

     The IO PEs distribution is done in a hopefully optimal way to maximize IO performance
     when using a large number of nodes.
     For output purposes, between 1 and 2 IO PEs per node is usually considered optimal.
     For input purposes one might use more PEs per node if the input operation involves
     heavvy computations (like horizontal interpolation).
     If the number of PEs in the X direction is pex and the number of PEs in the Y direction
     is pey, min(pex,pey)*min(pex,pey) is likely to be the maximum number of PEs in a set.
     The RPN_COMM_io_pe_valid_set subroutine (totally standalone) can be used to see if
     the requested IO PE set is possible given pex and pey.

rpn_comm/example example code [ Functions ]

[ Top ] [ Functions ]

EXAMPLE

  integer, external :: RPN_COMM_io_pe_test_callback   ! the callback function returns an integer value
  integer setno,nio,me_io,setno2,setno3,status
  integer, dimension(1), target :: argv               ! the argument list for the callback function
  integer, dimension(:,:), pointer :: iopelist        ! will receive a coordinate list
  integer, dimension(1) :: tbcst                      ! this will be broadcast in the test

EXAMPLE

  setno = RPN_COMM_create_io_set(nio+2,0)                   ! create a set of IO PEs containing nio+2 members
  me_io = RPN_COMM_is_io_pe(setno)                          ! is this PE a member of this IO PE set ?
  if(me_io .ne. -1) then
    print *,"I am a proud IO pe !"                          ! YES it is
  else
    print *,"I am a lazy  NON-IO pe !"                      ! NO it is not
  endif
  print *,"set number, size of set='",setno,RPN_COMM_io_pe_size(setno)         ! get size of this IO PE set
  setno2 = RPN_COMM_create_io_set(nio,0)                                       ! crete another set containing nio PEs
  print *,"set number, size of set='",setno2,RPN_COMM_io_pe_size(setno2)       ! get size of this other IO PE set (should be nio+2)
  setno = RPN_COMM_free_io_set(setno)                                          ! delete IO set
  print *,'DEBUG: freed IO set ',setno
  print *,"set number, size of set='",setno,RPN_COMM_io_pe_size(setno)         ! this should return -1
  setno = RPN_COMM_create_io_set(nio,0)                                        ! re create IO PE set, this time with nio PEs
  print *,"set number, size of set='",setno,RPN_COMM_io_pe_size(setno)         ! this should return nio now
  setno3 = RPN_COMM_create_io_set(nio-1,0)                                     ! crete another set containing nio-1 PEs
  print *,"set number, size of set='",setno3,RPN_COMM_io_pe_size(setno3)       ! this should return nio-1
  argv(1) = pe_me                                                              ! argument array for callback function
  status = RPN_COMM_io_pe_callback(setno3,RPN_COMM_io_pe_test_callback,C_LOC(argv(1)))    ! call to callback function(see example below)
  print *,"after callback, status,argv=",status,argv(1)                        ! status 0 from non members of IO set, return of callback function on members
  iopelist => RPN_COMM_io_pe_coord(setno3)                                     ! get grid coordinates of PEs in IO set setno3
  print *,"PE list x=",iopelist(:,1)                                           ! row 1, X coordinates in "GRID"
  print *,"PE list y=",iopelist(:,2)                                           ! row 2, Y coordinates in "GRID"
  tbcst = pe_me
  if(RPN_COMM_is_io_pe(setno3) .ne. -1)then  ! part of the set only, bcst pe_me of highest rank in set
    call RPN_COMM_io_pe_bcast(tbcst,1,'MPI_INTEGER',RPN_COMM_io_pe_size(setno3)-1,setno3,status) 
!   root for broadcast is RPN_COMM_io_pe_size(setno3)-1, get "GRID" ordinal of PE with highest rank (last PE) in IO set
!   root for broadcast 0 would have returned the "GRID" ordinal of first (lowest rank) PE in set
    print *,'tbcst after broadcast',tbcst                                      ! and the answer is ...
  endif

rpn_comm/example_callback example of callback function [ Functions ]

[ Top ] [ Functions ]

EXAMPLE

function RPN_COMM_io_pe_test_callback(argv,setno,comm,me,x,y,npe) result(status)  ! demo callback function
! this function will only be called on members of the IO PE set (setno)
  use ISO_C_BINDING
  implicit none
  integer, intent(IN) :: setno,comm,me,npe      ! set number, MPI communicator of this IO PE set, this PE's ordinal in set
  integer, dimension(npe), intent(IN) :: x,y
  type(C_PTR), intent(IN) :: argv               ! "blind pointer" to argument list
!
  integer :: status
  integer, dimension(:), pointer :: argvf        ! what the "blind pointer" will be mapped into
!
  call C_F_POINTER(argv,argvf,[1])                                ! transform C pointer into Fortran pointer
  print *,"IO PE CALLBACK set, me, argument = ",setno,me,argvf(1) ! an integer array of dimension 1 was expected
  print *,"CALLBACK x=",x                                         ! list of X "GRID" coordinates
  print *,"CALLBACK y=",y                                         ! list of Y "GRID" coordinates
  argvf(1) = -(100 +argvf(1))                                     ! we modify the argument list (optional)
  status = -123                                                   ! status to be returned to caller on this PE
  return
end

rpn_comm/RPN_COMM_create_io_set [ Functions ]

[ Top ] [ Functions ]

SYNOPSIS

function RPN_COMM_create_io_set(npes,method) result(setno)    ! create a set of IO PEs

ARGUMENTS

  integer, intent(IN)  :: npes        ! number of IO PEs desired in set ( set size will be min(npes,pe_nx,pe_ny) )
  integer, intent(IN)  :: method      ! method 0 is the only one supported for the time being
  integer :: setno                    ! set number created ( -1 if creation failed )

AUTHOR

  M.Valin Recherche en Prevision Numerique 2015

rpn_comm/RPN_COMM_free_io_set [ Functions ]

[ Top ] [ Functions ]

SYNOPSIS

function RPN_COMM_free_io_set(setno) result(freed)    ! delete a set of IO PEs created by RPN_COMM_create_io_set

ARGUMENTS

  integer, intent(IN) :: setno          ! set number as returned by RPN_COMM_create_io_set
  integer :: freed                      ! setno if operation succeeded, -1 if it failed

AUTHOR

  M.Valin Recherche en Prevision Numerique 2015

rpn_comm/RPN_COMM_io_pe_bcast [ Functions ]

[ Top ] [ Functions ]

SYNOPSIS

subroutine RPN_COMM_io_pe_bcast(buffer,count,datatype,root,setno,ierr) ! cannot easily publish interface because of buffer

ARGUMENTS

  integer, intent(IN) :: setno                ! set number as returned by RPN_COMM_create_io_set
  integer, intent(IN) :: count                ! number of elements of type datatype
  integer, intent(INOUT), dimension(*) :: buffer   ! data to be broadcast
  character (len=*), intent(IN) :: datatype   ! datatype RPN_COMM style
  integer, intent(IN) :: root                 ! root PE in set for broadcast (typically 0)
  integer, intent(OUT) :: ierr                ! status upon completion

AUTHOR

  M.Valin Recherche en Prevision Numerique 2015

rpn_comm/RPN_COMM_io_pe_callback [ Functions ]

[ Top ] [ Functions ]

SYNOPSIS

function RPN_COMM_io_pe_callback(setno,callback,argv) result(status)

ARGUMENTS

  integer, intent(IN) :: setno          ! set number as returned by create_ioset
  type(C_PTR) :: argv                   ! "pickled" arguments to callback
  integer, external :: callback         ! user routine to be called
  integer :: status

Notes

 callback to be executed only on members of an IO PE set (usually called by all PEs on grid)
 if this PE is not a member of the IO set, nothing happens, status = 0
 if the IO set is invalid, nothing happens, status = -1
 if this PE is a member of the set, callback is called and its return value is put into status

 the calling sequence of callback is as follows
 status = callback(argv,setno,comm,ordinal,x,y,npe)
 type(C_PTR) :: argv 
 integer :: setno         ! IO PE set number
 integer :: comm          ! communicator used by this IO PE set
 integer :: ordinal       ! ordinal of this PE in the set
 integer, dimension(npe) :: x  ! x grid coordinates for IO PEs in this set
 integer, dimension(npe) :: y  ! y grid coordinates for IO PEs in this set
 integer :: npe           ! number of PEs in this set

AUTHOR

  M.Valin Recherche en Prevision Numerique 2015

rpn_comm/RPN_COMM_io_pe_comm [ Functions ]

[ Top ] [ Functions ]

SYNOPSIS

function RPN_COMM_io_pe_comm(setno) result(communicator)     ! get communicator of IO set setno

ARGUMENTS

  integer, intent(IN) :: setno          ! set number as returned by RPN_COMM_create_io_set
  integer :: communicator               ! communicator for IO PE set if a member, null communicator otherwise

AUTHOR

  M.Valin Recherche en Prevision Numerique 2015

rpn_comm/RPN_COMM_io_pe_coord [ Functions ]

[ Top ] [ Functions ]

SYNOPSIS

function RPN_COMM_io_pe_coord(setno) result(list)     ! get x and y coordinates in grid of PEs in IO set setno

ARGUMENTS

  integer, intent(IN) :: setno                  ! set number as returned by RPN_COMM_create_io_set
  integer, dimension(:,:), pointer :: list      ! list of IO PE set, null pointer if set is not valid

AUTHOR

  M.Valin Recherche en Prevision Numerique 2015

rpn_comm/RPN_COMM_io_pe_gridid [ Functions ]

[ Top ] [ Functions ]

SYNOPSIS

function RPN_COMM_io_pe_gridid(setno,n) result(ordinal)     ! ordinal in "grid" of PE n of IO set setno

ARGUMENTS

  integer, intent(IN) :: setno         ! set number as returned by RPN_COMM_create_io_set
  integer, intent(IN) :: n             ! ordinal of PE for which ordinal in grid is sought (ORIGIN 0)
  integer :: ordinal                   ! ordinal in "grid" if n not larger than IO set size, -1 otherwise

AUTHOR

  M.Valin Recherche en Prevision Numerique 2015

rpn_comm/RPN_COMM_io_pe_groups [ Functions ]

[ Top ] [ Functions ]

SYNOPSIS

function RPN_COMM_io_pe_groups(setno) result(ngroups)     ! get number of groups in IO set setno

ARGUMENTS

  integer, intent(IN) :: setno          ! set number as returned by RPN_COMM_create_io_set
  integer :: ngroups                    ! ngroups of IO PE set, -1 if set is not valid

AUTHOR

  M.Valin Recherche en Prevision Numerique 2015

rpn_comm/RPN_COMM_io_pe_idlist [ Functions ]

[ Top ] [ Functions ]

SYNOPSIS

function RPN_COMM_io_pe_idlist(setno) result(idlist)     ! get list of PE ordinals in IO set setno

ARGUMENTS

  integer, intent(IN) :: setno                  ! set number as returned by RPN_COMM_create_io_set
  integer, dimension(:), pointer :: idlist        ! list of IO PE set, null pointer if set is not valid

AUTHOR

  M.Valin Recherche en Prevision Numerique 2015

rpn_comm/RPN_COMM_io_pe_size [ Functions ]

[ Top ] [ Functions ]

SYNOPSIS

function RPN_COMM_io_pe_size(setno) result(population)     ! get size of IO set setno

ARGUMENTS

  integer, intent(IN) :: setno          ! set number as returned by RPN_COMM_create_io_set
  integer :: population                 ! population of IO PE set, -1 if set is not valid

AUTHOR

  M.Valin Recherche en Prevision Numerique 2015

rpn_comm/RPN_COMM_io_pe_valid_set [ Functions ]

[ Top ] [ Functions ]

SYNOPSIS

function RPN_COMM_io_pe_valid_set(x,y,npes,penx,peny,diag,method) result(status)       
!  check if this IO PE configuration is possible

ARGUMENTS

    integer, dimension(npes), intent(OUT) :: x    ! x coordinates of PEs in set
    integer, dimension(npes), intent(OUT) :: y    ! y coordinates of PEs in set
    integer, intent(IN) :: npes                   ! number of PEs in set
    integer, intent(IN) :: penx, peny             ! number of PEs along x and y in PE grid
    integer, intent(IN) :: method                 ! fill method
    logical, intent(IN) :: diag                   ! if .true. print IO PE map and diagnostics
    integer :: status                             ! RPN_COMM_OK or RPN_COMM_ERROR

AUTHOR

  M.Valin Recherche en Prevision Numerique 2015

Notes

   some error messages are printed in case of error
   diagnostics include a map of IO PEs

rpn_comm/RPN_COMM_is_io_pe [ Functions ]

[ Top ] [ Functions ]

SYNOPSIS

function RPN_COMM_is_io_pe(setno) result(ordinal)     ! is this PE part of IO set setno ?

ARGUMENTS

  integer, intent(IN) :: setno         ! set number as returned by RPN_COMM_create_io_set
  integer :: ordinal                   ! ordinal in IO PE set if a member, -1 if not

AUTHOR

  M.Valin Recherche en Prevision Numerique 2015

rpn_comm/RPN_COMM_is_valid_io_setno [ Functions ]

[ Top ] [ Functions ]

SYNOPSIS

function RPN_COMM_is_valid_io_setno(setno) result(valid)     ! is this IO set number valid ?

ARGUMENTS

  integer, intent(IN) :: setno         ! set number as returned by RPN_COMM_create_io_set
  logical :: valid                     ! .true. if set is valid, .false. otherwise

AUTHOR

  M.Valin Recherche en Prevision Numerique 2015