perhaps the equations utilized for calculating gas and liquid fugacities are available in the operating manual or technical literature related to your software,
anyway you may find a detailed discussion for different EOS in The Properties of Gases and Liquids by Reid, Prausnitz, Poling and other textbooks which may be available at your university library,
about the isothermal flash itself, it can be solved in different ways,
as noted by Art developers prefer do not share the details,
however a simple method to solve a isothermal vapor-liquid equilibria is the Successive Substitution Method (SSM) with many variants discussed by Michelsen and other authors, you may find examples (as EXCEL pages, FORTRAN codes etc.) with google,
things are more difficult in critical area (with strong composition's dependence) or multiphase equilibria,
in multiphase equilibria you can solve for a different number of phases and states (vapor, liquid, solid) at equilibria,
since this is a student's forum I wish to mention that there are free versions of tools which do that, for example Prode Properties which I utilize,
when coding your procedure the first step should be the methods for calculating fugacities of vapor and liquid phases,
there are Fortran codes by Michelsen and other authors which can help,
you can test the calculated values against those of a software which exports fugacities and derivatives (see for example Prode Properties)