logo
down
shadow

Can't open a to big .csv file in python


Can't open a to big .csv file in python

By : user2949147
Date : November 16 2020, 06:23 AM
I wish this helpful for you You can't "split a file", but you can read it line by line no matter how big. E.g:
code :
import collections

btcDatear = []
btcPricear = []
btcVolumear = []
howfarback = 20000
try:
    with open('.btceUSD.csv', 'r') as sourceCode:
        lastNlines = collections.deque(sourceCode, howfarback)
    for eachline in lastNlines:
        splitLine = eachline.split(',')
        btcDate = splitLine[0]
        btcPrice = splitLine[1]
        btcVolume = splitLine[2]

        btcDatear.append(float(btcDate))
        btcPricear.append(float(btcPrice))
        btcVolumear.append(float(btcVolume))
except Exception as e:
    print "failed raw data", str(e)
    for fields in csv.reader(iter(lastNlines)):
        btcDate, btcPrice, btcVolume = fields[:3]


Share : facebook icon twitter icon
Python file open/close every time vs keeping it open until the process is finished

Python file open/close every time vs keeping it open until the process is finished


By : Jonathan Lewis
Date : March 29 2020, 07:55 AM
wish of those help You should definitely try to open/close the file as little as possible
Because even comparing with file read/write, file open/close is far more expensive
code :
f=open('test1.txt', 'w')
for i in range(1000):
    f.write('\n')
f.close()
for i in range(1000):
    f=open('test2.txt', 'a')
    f.write('\n')
    f.close()
Why I can't read anything with File.open in python when I open the file to write first?

Why I can't read anything with File.open in python when I open the file to write first?


By : Clint Mock
Date : March 29 2020, 07:55 AM
help you fix your problem , You never closed the file with this line of code:
code :
f.close
with open('day_temps.txt', 'w') as handle:
    handle.write("10.3,10.1,9.9,9.9,9.8,9.6,9.0,10.1,10.2,11.1")
with open('day_temps.txt', 'w') as handle:
    handle.write("10.3,10.1,9.9,9.9,9.8,9.6,9.0,10.1,10.2,11.1")

def get_stats(file_name):
    with open(file_name, 'r') as handle:
        numbers = map(float, handle.read().split(','))

    return min(numbers), max(numbers), sum(numbers) / len(numbers)

if __name__ == '__main__':
    stats = get_stats('day_temps.txt')
    print "({0:.5}, {1:.5}, {2:.5})".format(*stats)
Python: when writing to a large file, keep the file open or to open it and append to it as needed?

Python: when writing to a large file, keep the file open or to open it and append to it as needed?


By : Devon936
Date : March 29 2020, 07:55 AM
To fix the issue you can do Opening and closing the files definitely has a cost. However if your legacy program takes one or more second to respond you propably won't notice.
code :
def func1():
    for x in range(1000):
        x = str(x)
        with open("test1.txt", "a") as k:
            k.write(x)

1 loops, best of 3: 2.47 s per loop

def func2():
    with open("test2.txt", "a") as k:
        for x in range(1000):
            x = str(x)
            k.write(x)

100 loops, best of 3: 6.66 ms per loop
def func3(file):
    for x in range(10):
        x = str(x)
        with open(file, "a") as k:
            k.write(x)
10 loops, best of 3: 33.4 ms per loop
1 loops, best of 3: 24.5 s per loop
Python PIL - open image from file object: File not open for reading

Python PIL - open image from file object: File not open for reading


By : benny
Date : November 14 2020, 04:51 PM
hop of those help? You opened the file for writing, not reading. You'd have to use a dual mode, and first rewind the file pointer:
code :
with open('myimage.png', 'w+b') as imgfile:
    imgfile.write(decodestring(base64_image))
    imgfile.seek(0)
    f = Image.open(imgfile)
from io import BytesIO

imgfile = BytesIO(decodestring(base64_image))
f = Image.open(imgfile)
Pass a file to Python via the Right-Click 'Open With' context menu in windows, to then open in another program

Pass a file to Python via the Right-Click 'Open With' context menu in windows, to then open in another program


By : user3120510
Date : March 29 2020, 07:55 AM
fixed the issue. Will look into that further You can access command line arguments passed to your script using sys.argv. As long as you want to pass these arguments to some third-party application(which is an executable binary) you should use the subprocess module as os.startfile is meant to start a file with its associated application.
code :
import sys
import subprocess

def main():
    if len(sys.argv) == 1:
        path = sys.argv[0]
        subprocess.run(['/path/to/my.exe', path])
    else:
        print('Usage myscript.py <path>')

if __name__ == "__main__":
    main()
Related Posts Related Posts :
  • Return new instance of subclass when using methods inherited from parent class in Python
  • Which function in django.contrib.auth creates the default model permissions?
  • Formatting text in tabular form with Python
  • How to determine the first day of a month in Python
  • Error while converting date to timestamp in python
  • Python string iterations
  • Is there any file number limitation when you select multiple files with wxFileDialog?
  • Errors with Matplotlib when making an executable with Py2exe (Python)
  • Django Haystack - Indexing single field
  • Go Pro Hero 3 - Streaming video over wifi
  • Appending a column in .csv with Python/Pandas
  • How to change my result directory in Robot framework using RIDE?
  • problem with using pandas to manipulate a big text file in python
  • python-magic module' object has no attribute 'open'
  • Where goes wrong for this High Pass Filter in Python?
  • Why inserting keys in order into a python dict is faster than doint it unordered
  • flann index saving in python
  • Create new instance of list or dictionary without class
  • How can I easily convert FORTRAN code to Python code (real code, not wrappers)
  • Address of lambda function in python
  • Python adding space between characters in string. Most efficient way
  • python http server, multiple simultaneous requests
  • Disguising username & password on distributed python scripts
  • Post GraphQL mutation with Python Requests
  • Why doesnt pandas create an excel file?
  • Rolling comparison between a value and a past window, with percentile/quantile
  • How to avoid repetitive code when defining a new type in python with signature verification
  • How to configure uWSGI in order to debug with pdb (--honour-stdin configuration issue)
  • In Python, how do you execute objects that are functions from a list?
  • Python- Variable Won't Subtract?
  • Processing Power In Python
  • Python 2.7.2 - Cannot import name _random or random from sys
  • Why doesn't the Python sorted function take keyword order instead of reverse?
  • Make a function redirect to other functions depending on a variable
  • get_absolute_url in django-categories
  • Monitoring non-Celery background task with New Relic in Python
  • Feature selection with LinearSVC
  • LSTM - Predicting the same constant values after a while
  • Test the length of elements in a list
  • Django: render radiobutton with 3 columns, cost column must change according to size & quantity selected
  • Python class attributes vs global variable
  • sys.stdout.writelines("hello") and sys.stdout.write("hello")
  • is ndarray faster than recarray access?
  • Python - search through directory trees, rename certain files
  • GAE: How to build a query where a string begins with a value
  • TypeError: __init__() takes at least 2 arguments (1 given)
  • Overriding and customizing "django.contrib.auth.views.login"
  • Django : Redirect to a particular page after login
  • Python search and copy files in directory
  • pretty printing numpy ndarrays using unicode characters
  • Frequent pattern mining in Python
  • How can I make a set of functions that can be used synchronously as well as asynchronously?
  • Convert one dice roll to two dice roll
  • count occourrence in a list
  • Writing an If condition to filter out the first word
  • to read file and compare column in python
  • Install python-numpy in the Virtualenv environment
  • `.select_by_visible_text()` is failed to select element?
  • Unable to send data multiple requests in a single connection — socket error
  • Pandas HDFStore unload dataframe from memory
  • shadow
    Privacy Policy - Terms - Contact Us © ourworld-yourmove.org