Next generation networks are expected to support large volume of data tra c generatedrnfrom emerging applications such as ultra high speed video streaming, machine-tomachinern(M2M) communication and the Internet of things (IoT). To handle this largernvolume of data tra c, these networks should employ technologies that utilize broad spectrum,rno er higher cell density and high spectral e ciency. The spectral e ciency can bernimproved by increasing the transmitter power; introducing additional processing (such asrndeploying multiple antenna systems and advanced modulation techniques) in transceiverrnpairs that help to harvest energy; minimize multiuser-interference; and implementing innovativernwireless planning and operation strategies that save energy.rnBy deploying very large numbers of antennas at a base station (BS), which is calledrnmassive multiple input multiple output (MIMO), we can signi cantly improve the spectralrne ciency of mobile networks. Besides, massive MIMO simpli es transmission processing,rnimproves energy e ciency and reduces the required transmission power of the users. Duernto those performance gains, massive MIMO has become an enabler for the deployment ofrn5G and beyond networks.rnIn this PhD research, we study and analyze channel modeling, resource allocation andrnoptimization techniques in massive MIMO systems. For this, rst we analyze recent worksrnon signal processing, channel modeling, channel estimation, resource allocation and optimizationrntechniques in massive MIMO systems. In this regard, fundamentals of massivernMIMO systems including channel capacity, spectral e ciency and energy e ciency havernbeen studied. Closedform lower bound expressions are derived for the spectral e ciencyrnand energy e ciency. Then, simulation results are provided to validate the theoretical analysis.rnBesides, performance analysis is done for linear detection and precoding techniquesrnand then computationally e cient inverse approximation techniques are proposed for linearrndetection and precoding in massive MIMO systems. Speci cally, Truncated Neumannrnseries-based matrix inversion approximation techniques are formulated and probability of convergence, error of approximation and computationally complexity are analyzed.rnThen, we analyze achievable spectral e ciency of massive MIMO systems in realisticrnpropagation environment under perfect and imperfect channel state information (CSI) scenarios.rnIn particular, the e ects of major large scale and small fading parameters includingrnpathloss, shadowing, multipath fading, spatial channel correlation and impact of channelrnestimation have been investigated. Spectral e ciency analysis is done for uplink massivernMIMO system under Rician fading channel model. Besides, by applying non-central torncentral Wishart approximation, closedform lower bound achievable rate expressions arernformulated for massive MIMO systems in Rican fading channel model.rnThen, energy e cient power control and resource allocation algorithms have been proposed.rnFor this, rst by using large system analysis, analytical closedform lower boundrnexpressions are derived for the achievable sum rate and appropriate power consumptionrnmodel is formulated for the proposed massive MIMO systems. Then, by utilizing tools fromrnfractional programming theory and sequential convex programming, energy e cient powerrncontrol and resource allocation algorithms have been formulated. Further, the impacts ofrnsystem and propagation parameters on energy e ciency have been evaluated. Particularly,rnthe impacts of maximums transmitter power and minimum rate constraints of the usersrnon global energy e ciency have been evaluated. The results show that the global energyrne ciency increases with the maximum transmitter power constraint and decreases withrnthe minimum data rate constraint.rnFinally, we analyze the performance of multicell massive MIMO systems in spatiallyrncorrelated channel model. First, we study and evaluate fundamentals of multicell massivernMIMO systems. In this regard channel modeling, power allocation and spatial resourcernallocation in multicell massive MIMO systems are considered. Besides, we evaluate thernimpacts of spatial correlation and pilot contamination. Important trade-o s and considerationsrnon design and optimization of multicell massive MIMO systems has been studied.rnThe impacts of system and propagation parameters are evaluated theoretically and viarnnumerical simulation. The results show that spatial channel correlation has a major impactrnon channel hardening, favorable propagation, channel estimation quality and spectralrne ciency of the system.